Solves TOD systems' query annotation problem without explicit annotations.
problem Training TOD systems without explicit KB query annotation.
method Reinforcement learning (RL) and pipelined approach for query prediction and system training.
result Improved RL agent with modifications for TOD tasks.
We consider the problem of prediction by a machine learning algorithm, called learner, within an adversarial learning setting. The learner's task is to correctly predict the class of data passed to it as a query. However, along with queries containing clean data, the learner could also receive malicious or adversarial …
Optimizes state monitoring in Markovian systems with cost constraints.
problem Balancing state queries with prediction costs in Markovian systems.
method Greedy policy and SGD-based learning variant for optimal predict-query tradeoff.
result Greedy policy is suboptimal but performs close to optimal under certain conditions.
C-IP improves LLMs' query selection for interactive tasks by estimating uncertainty robustly.
problem Minimizing the number of queries for interactive LLMs.
method Conformal Information Pursuit (C-IP) using conformal prediction sets.
result C-IP achieves better predictive performance and shorter query-answer chains.
We model GitHub interactions as a temporal knowledge graph for software engineering questions.
problem Insufficient performance of existing temporal models on extrapolated queries and time prediction.
method Introduced an extension to current temporal models using relative temporal information.
result Improved performance on extrapolated queries and time prediction.
New insights on active sequential prediction for mean estimation.
problem Active sequential prediction-powered mean estimation problem.
method Combining uncertainty-based suggestion with a constant probability, analyzing non-asymptotic bounds, and using no-regret learning.
result The optimal query probability is close to the constraint when using no-regret learning.
Many real-world problems can be formalized as predicting links in a partially observed network. Examples include Facebook friendship suggestions, consumer-product recommendations, and the identification of hidden interactions between actors in a crime network. Several link prediction algorithms, notably those recently …
Assessing the predictive accuracy of black box classifiers is challenging in the absence of labeled test datasets. In these scenarios we may need to rely on a human oracle to evaluate individual predictions; presenting the challenge to create query algorithms to guide the search for points that provide the most informa…
Attention improves edge prediction in e-commerce graphs.
problem Predicting edges in graphs from node attributes in e-commerce.
method Used attention mechanism in simple feedforward networks, derived analytically tractable model AttEST.
result Attention network outperforms LSTM architectures by over 20% on F-1 score.
StreamEnsemble dynamically selects ML models for ST data streams to improve predictive accuracy.
problem Predictive queries over spatiotemporal data streams are challenging due to varying distributions and patterns.
method Dynamic selection and allocation of ML models based on time series distributions and characteristics.
result Significantly outperforms traditional ensemble and single model approaches, reducing prediction error by over 10 times.
Study efficient sequential evaluation of large language models using historical data.
problem Sequentially evaluate a new large language model (LLM) on a fixed question set.
method Construct a confidence sequence (CS) and design active querying rules to shrink CS width.
result Simple uniform sampling can sometimes outperform adaptive querying rules.
LEAQI learns to query an expert less often for noisy guidance.
problem Active imitation learning with noisy guidance to reduce expert query complexity.
method Developed LEAQI, a difference classifier that predicts expert disagreement with noisy heuristic.
result Significantly fewer queries to the expert with comparable or better accuracy.
Although interactive learning puts the user into the loop, the learner remains mostly a black box for the user. Understanding the reasons behind queries and predictions is important when assessing how the learner works and, in turn, trust. Consequently, we propose the novel framework of explanatory interactive learning…
The goal of query performance prediction (QPP) is to automatically estimate the effectiveness of a search result for any given query, without relevance judgements. Post-retrieval features have been shown to be more effective for this task while being more expensive to compute than pre-retrieval features. Combining mult…
Bayesian optimization enhanced with conformal prediction for better outcome reliability.
problem Uncertainty and model misspecification in Bayesian optimization.
method Conformal prediction to provide coverage guarantees and Bayesian optimization to select queries.
result Significant improvement in query coverage without sacrificing sample-efficiency.
Proposes CRA framework for certifying fair predictive models.
problem Certifying fairness of predictive models trained on biased data.
method Formulates CRA for fairness queries, uses background knowledge and limited target population stats.
result Builds certifiably fair predictive models on target population.
A new method for making interpretable predictions by sequentially asking questions, faster and more efficient.
problem Developing interpretable machine learning models for complex tasks.
method Variational Information Pursuit (V-IP) that bypasses the need for learning generative models.
result V-IP is 10-100x faster and finds shorter query chains compared to IP and reinforcement learning.
Budgeted deferral framework reduces expert query costs in machine learning.
problem High cost of querying all experts in deferral algorithms.
method Introduces budgeted deferral framework for selective expert querying.
result Empirical results show substantial reduction in training costs without sacrificing accuracy.
Paper studies how incomplete data affects machine learning, proposing 'Certain Predictions' for NN classifiers.
problem Impact of incomplete and inconsistent data on machine learning applications.
method Extends Certain Answers concept to machine learning, introduces 'Certain Predictions' for NN classifiers.
result Developed efficient solutions for CP queries in NN classifiers, improving classification accuracy.
CUBE explains models by balanced experiments and contrasts.
problem Post-hoc explanation of trained predictive models.
method Design-based framework using balanced low-high probes.
result Reveals dominant learned effect structure and clarifies query efficiency.
Adaptive querying learns user psychometrics with AI personas.
problem Learning user psychometrics within query budgets.
method Persona-induced latent variable model with AI personas and large language model response distributions.
result Persona-based posteriors deliver accurate probabilistic predictions.
Learning low-dimensional embeddings of knowledge graphs is a powerful approach used to predict unobserved or missing edges between entities. However, an open challenge in this area is developing techniques that can go beyond simple edge prediction and handle more complex logical queries, which might involve multiple un…
Proposes a query-efficient blackbox attack method.
problem Adversarial attacks on machine learning models, especially deep neural networks.
method QEBA: Query-Efficient Boundary-based blackbox Attack, using only final prediction labels.
result QEBA achieves 100% attack success rate with fewer queries and lower perturbation.
Proposes a new query autocompletion method that maximizes retrieval performance.
problem Users often select suboptimal queries due to unknown best retrieval performance.
method Formulates query autocompletion as ranking item rankings, uses counterfactual learning.
result Empirical results show improved query suggestions for better retrieval performance.
Bayesian framework for online consensus prediction from expert feedback.
problem Online classification with expert consensus prediction, cost-effective.
method General Bayesian framework for dynamic expert consensus estimation.
result Demonstrated superior performance on large-scale crowdsourced datasets.
Marich extracts high-fidelity models from public data with minimal queries.
problem Creating an accurate replica of a target ML model using few queries.
method Sequentially selects informative queries to maximize entropy and reduce model mismatch.
result Extracted models achieve 60-95% of target model's accuracy with 1,000-8,500 queries.
Improves model classification accuracy in black-box settings.
problem Difficulty in inferring model properties due to limited query access.
method Introduces discriminative factorization to distinguish high-quality queries.
result Probability of chance-level classification decreases exponentially with query budget.
A keyword spotting (KWS) system determines the existence of, usually predefined, keyword in a continuous speech stream. This paper presents a query-by-example on-device KWS system which is user-specific. The proposed system consists of two main steps: query enrollment and testing. In query enrollment step, phonetic pos…
Predicting the click-through rate of an advertisement is a critical component of online advertising platforms. In sponsored search, the click-through rate estimates the probability that a displayed advertisement is clicked by a user after she submits a query to the search engine. Commercial search engines typically rel…
Solving sequential decision prediction problems, including those in imitation learning settings, requires mitigating the problem of covariate shift. The standard approach, DAgger, relies on capturing expert behaviour in all states that the agent reaches. In real-world settings, querying an expert is costly. We propose …
Study learns linear utility functions from comparisons, showing learnability gaps between passive and active learning.
problem Learn linear utility functions from pairwise comparison queries.
method Analyzes passive and active learning settings, considering noise-free and noisy query responses.
result Efficient learnability of linear utilities in passive learning, but not for utility parameters without strong assumptions.
Improves search performance by transferring knowledge from recommender system.
problem Cold start and feedback loop problems in search retrieval.
method Zero-Shot Heterogeneous Transfer Learning framework.
result Significant improvements in relevance and user interactions over production system.
Deep Neural Networks, and specifically fully-connected convolutional neural networks are achieving remarkable results across a wide variety of domains. They have been trained to achieve state-of-the-art performance when applied to problems such as speech recognition, image classification, natural language processing an…
Efficient methods for answering complex probabilistic queries in sequential data.
problem Complex probabilistic queries in sequential data.
method Broad class of novel approximation techniques for marginalization in sequential models.
result Efficient techniques for answering long-range probabilistic queries.
The paper bridges theory and practice in query-driven selectivity learning.
problem Insufficient theoretical understanding of query-driven selectivity learning.
method Demonstrates learnability of selectivity predictors and establishes favorable OOD generalization error bounds.
result Theoretical advances improve OOD generalization of query-driven selectivity models.
Paper proposes a new pipeline for few-shot classification using forget-update module and channel vector sequence.
problem Few-shot classification with limited support samples.
method Channel vector sequence construction module and forget-update module.
result Pipeline achieves state-of-the-art results on various datasets.
We investigate how an adversary can optimally use its query budget for targeted evasion attacks against deep neural networks in a black-box setting. We formalize the problem setting and systematically evaluate what benefits the adversary can gain by using substitute models. We show that there is an exploration-exploita…
Optimal query allocation improves extractive QA efficiency with LLMs.
problem Efficiency and reliability in extractive question answering with LLMs.
method Learning-to-Defer framework that allocates queries to specialized models.
result Enhanced answer reliability with reduced computational overhead.
Recently, several studies have explored methods for using KG embedding to answer logical queries. These approaches either treat embedding learning and query answering as two separated learning tasks, or fail to deal with the variability of contributions from different query paths. We proposed to leverage a graph attent…
New algorithm for clustering with faulty oracle achieves optimal queries and efficiency.
problem Clustering with a faulty oracle, especially for multiple clusters.
method Built on stochastic block model, provides nearly-optimal query complexity.
result Time-efficient algorithm with nearly-optimal query complexity for all constant k and any δ.
This paper proposes LDM for efficient deep active learning.
problem Improving model performance by selecting informative unlabeled data.
method Least Disagree Metric (LDM) and its estimator for deep learning models.
result LDM-based active learning achieves state-of-the-art performance.
Transductive inference is an effective means of tackling the data deficiency problem in few-shot learning settings. A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples, or confidence-weighted average…
Optimizes query routing to LLMs under cost and resource constraints.
problem Non-uniform or adversarial batching in per-query routing methods leads to cost inefficiency.
method Batch-level, resource-aware routing framework that jointly optimizes model assignment for each batch.
result Robust routing framework improves accuracy by 1-14% over non-robust methods.
Statistical query algorithms and low-degree tests are nearly equivalent in high-dimensional hypothesis testing.
problem High-dimensional hypothesis testing and information-computation gaps.
method Analysis of statistical query framework and low-degree polynomials.
result Statistical query algorithms and low-degree polynomials are almost equivalent in power under mild conditions.
In this paper, we initiate a rigorous theoretical study of clustering with noisy queries (or a faulty oracle). Given a set of n elements, our goal is to recover the true clustering by asking minimum number of pairwise queries to an oracle. Oracle can answer queries of the form : "do elements u and v belong to the…
To access data stored in relational databases, users need to understand the database schema and write a query using a query language such as SQL. To simplify this task, text-to-SQL models attempt to translate a user's natural language question to corresponding SQL query. Recently, several generative text-to-SQL models …
Paper introduces efficient methods for probabilistic querying of event sequences.
problem Hard queries about future events in continuous-time sequences.
method Importance sampling framework for addressing query types.
result Method is more efficient than naive simulation, often 1,000 times.
The DEBS Grand Challenge 2018 is set in the context of maritime route prediction. Vessel routes are modeled as streams of Automatic Identification System (AIS) data points selected from real-world tracking data. The challenge requires to correctly estimate the destination ports and arrival times of vessel trips, as ear…