PROPEL learns interpretable programmatic policies using imitation and projection.
problem Learning interpretable programmatic policies in reinforcement learning.
method PROPEL is a meta-algorithm based on three insights: optimization in policy space, neural-program mixing, and imitation synthesis.
result PROPEL significantly outperforms state-of-the-art approaches in learning programmatic policies.
We present a reinforcement learning framework, called Programmatically Interpretable Reinforcement Learning (PIRL), that is designed to generate interpretable and verifiable agent policies. Unlike the popular Deep Reinforcement Learning (DRL) paradigm, which represents policies by neural networks, PIRL represents polic…
The paper tackles calibrating long-term behaviors with multiple styles using programmatic style-consistency.
problem Generating long-term sequential behaviors with multiple styles simultaneously.
method Leverage programmatic labeling functions to specify controllable styles and derive style-consistency as a learning objective.
result Learned policies can be calibrated for up to 1024 distinct style combinations.
Deep reinforcement learning has led to several recent breakthroughs, though the learned policies are often based on black-box neural networks. This makes them difficult to interpret and to impose desired specification constraints during learning. We present an iterative framework, MORL, for improving the learned polici…
Paper introduces methods for more reliable probabilistic predictions with confidence intervals.
problem Inaccurate labeling of datasets due to unreliable probabilistic predictions from weak labeling functions.
method Proposes a methodology to provide confidence intervals for label probabilities using uncertainty sets of distributions.
result Improves reliability of probabilistic predictions and provides confidence intervals for label probabilities.
Programmatic Motion Concepts learn human actions from paired videos.
problem Learning motion concepts from paired video and action sequences.
method Semi-supervised learning architecture for hierarchical motion representation.
result Outperforms established baselines, especially in small data settings.
Predicting click and conversion probabilities when bidding on ad exchanges is at the core of the programmatic advertising industry. Two separated lines of previous works respectively address i) the prediction of user conversion probability and ii) the attribution of these conversions to advertising events (such as clic…
We present the perceptor gradients algorithm -- a novel approach to learning symbolic representations based on the idea of decomposing an agent's policy into i) a perceptor network extracting symbols from raw observation data and ii) a task encoding program which maps the input symbols to output actions. We show that t…
FABLE incorporates instance features into PWS label models for improved performance.
problem Lack of instance features in existing label models limits their performance.
method FABLE uses a mixture of Bayesian label models and a Gaussian Process classifier to incorporate instance features.
result FABLE achieves the highest averaged performance across nine baselines on benchmark datasets.
Adapting neural networks to guide program optimization for better classifiers.
problem Learning differentiable programs with complex architectures.
method Formulating program optimization as a graph search problem, using neural networks as heuristic relaxations.
result Trained neural networks can guide combinatorial search for programmatic classifiers, improving accuracy and interpretability.
Improves labeling quality in machine learning with pairwise feedback.
problem Scalability and quality of labeled datasets in machine learning.
method Incorporates pairwise feedback into the programmatic creation of labeled datasets.
result Even a small number of pairwise feedback sources can substantially improve label quality.
Improves understanding of PWS by calculating influence of sources and data.
problem Understanding the influence of each component in PWS.
method Proposes source-aware Influence Function (IF) to decompose and calculate influence.
result Improves end model's generalization performance and identifies mislabeling.
Expands weak supervision by allowing partial labels from multiple noisy sources.
problem Creating models without labeled data using heuristic labelers.
method Probabilistic generative model estimating partial label accuracies.
result Improved model accuracy on various tasks (8.6% on text, comparable to zero-shot methods on images).
We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as offensive basketball gameplay. When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables. Furtherm…
Condensa programmatically optimizes neural network compression.
problem Finding optimal compression strategies for neural networks.
method Bayesian optimization-based algorithm for automatic sparsity inference.
result Significant memory and runtime improvements for real-world DNNs.
Improved learning algorithm for first-price auctions reduces regret significantly.
problem Challenges in learning optimal bidding strategies for first-price auctions.
method Introduced novel ideas to achieve lower regret in sequential learning.
result Achieved log2(T) regret when opponents' bid distribution is known, and T1/3+ε regret in learning case. Stablecoin system improves resilience to extreme market events.
problem Vulnerability of stablecoins to extreme volatility and adversarial attacks.
method MVF-Composer uses multi-agent simulations to stress-test and down-weight manipulative signals.
result Reduces peak peg deviation by 57% and mean recovery time by 3.1x under adversarial conditions.
Compound examines decentralized lending users and their short loan durations.
problem Systemic risk in decentralized finance due to concentration and interconnection.
method Analysis of on-chain transaction data and smart contract programming.
result Many users borrow for yield farming, not for traditional lending.
Generative AI improves stock selection by synthesizing features from diverse data sources.
problem Automating feature discovery in stock market data.
method Used large language models with retrieval-augmented generation and structured prompting to synthesize features from various data sources.
result AI-generated features consistently outperform baselines, with Sharpe improvements ranging from 14% to 91%.
Paper trains models to resist string transformations.
problem Vulnerability of NLP models to adversarial string transformations.
method Combines search and abstraction techniques for robust training.
result Trained models resist combinations of user-defined transformations.
The paper introduces metrics to objectively evaluate interpretability methods.
problem Lack of objective evaluation metrics for interpretability methods.
method Proposes a set of metrics to evaluate interpretability methods along simplicity and broadness.
result Validated metrics on different benchmark tasks and showed their utility in method selection.
BAT benchmark for autobidding tasks in RTB auctions.
problem Lack of comprehensive datasets and benchmarks for autobidding.
method Developed a benchmark for two auction formats, implemented robust baselines.
result Provides a framework for developing and refining autobidding algorithms.
Chemical representations derived from deep learning are emerging as a powerful tool in areas such as drug discovery and materials innovation. Currently, this methodology has three major limitations - the cost of representation generation, risk of inherited bias, and the requirement for large amounts of data. We propose…
New method estimates model performance bounds without ground truth labels.
problem Evaluation of weakly supervised models without direct access to ground truth labels.
method Formulates model evaluation as a partial identification problem and uses Fréchet bounds for performance estimation.
result Derives accurate and computationally efficient bounds for key metrics like accuracy, precision, recall, and F1-score.
Paper presents a model for identifying informative COVID-19 tweets.
problem Identifying informative COVID-19 tweets on Twitter.
method Leveraged transformers (RoBERTa, XLNet, BERTweet) trained in Semi-Supervised Learning (SSL) setting.
result Achieved F1 score of 0.9011 on test set, ranking 7th on leaderboard.
Labeling training datasets has become a key barrier to building medical machine learning models. One strategy is to generate training labels programmatically, for example by applying natural language processing pipelines to text reports associated with imaging studies. We propose cross-modal data programming, which gen…
Obtaining enough labeled data to robustly train complex discriminative models is a major bottleneck in the machine learning pipeline. A popular solution is combining multiple sources of weak supervision using generative models. The structure of these models affects training label quality, but is difficult to learn with…
Nemo improves WS learning pipeline by 20%.
problem Creating effective labeling heuristics for weak supervision.
method Interactive procedure for designing heuristics, strategic data selection, contextualization of heuristics.
result Improves WS learning pipeline by 20%.
This paper examines interest rates and market efficiency in DeFi loanable funds protocols.
problem Equilibrium of supply and demand for loanable funds in DeFi protocols.
method Review of interest rate mechanisms in Compound, Aave, and dYdX; empirical analysis of market efficiency and inter-connectedness.
result Interest rate rules in DeFi protocols do not always equilibrate supply and demand.
Paper proposes real-time risk metrics for stablecoin protocols.
problem Lack of risk management frameworks for stablecoins.
method Developed two risk metrics: capitalization and liquidity.
result Demonstrated practical benefits of real-time on-chain data.
Generative models enhance weak supervision for better image classification.
problem Lack of labeled data in supervised learning.
method Fusion of generative adversarial networks and weak supervision.
result Model improves multiclass image classification performance.
Reinforcement learning and planning methods require an objective or reward function that encodes the desired behavior. Yet, in practice, there is a wide range of scenarios where an objective is difficult to provide programmatically, such as tasks with visual observations involving unknown object positions or deformable…
DPBD simplifies labeling functions through interactive demonstrations.
problem Difficulty in writing labeling functions for large-scale labeled training data.
method Data Programming by Demonstration (DPBD) framework using interactive demonstrations.
result Ruler system generates labeling rules more easily and with higher user satisfaction.
New method speeds up Bayesian optimization in high dimensions.
problem High-dimensional expensive function optimization struggles.
method Structured automatic differentiation for kernel matrices.
result First-order Bayesian optimization scalable to high dimensions.
Rapid progress in deep learning has spurred its application to bioinformatics problems including protein structure prediction and design. In classic machine learning problems like computer vision, progress has been driven by standardized data sets that facilitate fair assessment of new methods and lower the barrier to …
Adapts GRPO for off-policy RL, improving reward.
problem Improving training stability and efficiency in RL.
method Adapts GRPO to off-policy setting, uses clipped surrogate objectives.
result Off-policy GRPO outperforms on-policy GRPO in empirical tests.
Paper tackles efficient evaluation of natural stochastic policies in offline RL.
problem Efficiency issues in evaluating natural stochastic policies due to unknown evaluation policy.
method Derive efficiency bounds for tilting and modified treatment policies, propose nonparametric estimators.
result Proposed estimators attain efficiency bounds under lax conditions and enjoy partial double robustness.
New framework studies policy learning problems under data scarcity.
problem Learning improving policies when data is insufficient.
method Developed a mathematical framework for policy learning problems.
result Reduced policy learning problems to simpler ones in sample complexity.
New algorithms improve policy evaluation in reinforcement learning.
problem Off-policy stability and on-policy efficiency issues in policy evaluation.
method Introduced novel algorithms using oblique projection method.
result Demonstrated both off-policy stability and on-policy efficiency.
We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method. Prior off-policy policy gradient approaches have generally ignored the mismatch between the distribution of states visited under the behavior policy used to collect data, and what …
Novel approach trains LLMs for inductive reasoning using probabilistic programs.
problem Training LLMs for inductive reasoning with sparse, ambiguous data.
method Program-based Posterior Training (PPT) using probabilistic inference.
result Significant improvement in estimation accuracy and alignment with human judgments.
Large labeled training sets are the critical building blocks of supervised learning methods and are key enablers of deep learning techniques. For some applications, creating labeled training sets is the most time-consuming and expensive part of applying machine learning. We therefore propose a paradigm for the programm…
Stabilizes policy optimization with off-policy data using divergence augmentation.
problem Premature convergence and instability in policy optimization with off-policy data.
method Incorporates Bregman divergence between behavior and current policies to ensure safe policy updates.
result Empirically shows better performance in data-scarce scenarios compared to other algorithms.
New method estimates state-action stationary distribution for better off-policy policy evaluation.
problem Accurately estimating state-action stationary distribution for off-policy policy evaluation.
method Estimated Mixture Policy (EMP) for state and state-action stationary distribution corrections.
result Empirical validation shows improved accuracy over state-of-the-art methods.
Deep neural networks work well at approximating complicated functions when provided with data and trained by gradient descent methods. At the same time, there is a vast amount of existing functions that programmatically solve different tasks in a precise manner eliminating the need for training. In many cases, it is po…
Probabilistic programs with dynamic computation graphs can define measures over sample spaces with unbounded dimensionality, which constitute programmatic analogues to Bayesian nonparametrics. Owing to the generality of this model class, inference relies on `black-box' Monte Carlo methods that are often not able to tak…
We consider the problem of off-policy evaluation in Markov decision processes. Off-policy evaluation is the task of evaluating the expected return of one policy with data generated by a different, behavior policy. Importance sampling is a technique for off-policy evaluation that re-weights off-policy returns to account…
New methods estimate policy value and gradients for deterministic policies from off-policy data.
problem Estimating policy value and gradients for deterministic policies from off-policy data.
method Proposed new doubly robust estimators based on kernelization approaches.
result Demonstrated a rate independent of horizon length for policy value and gradient estimation.