FactTest assesses LLM factuality with Type I error control.
problem Lack of rigorous factuality verification for LLMs.
method Formulates factuality testing as hypothesis testing, ensuring Type I and II error control.
result Improves model accuracy by over 40% in abstaining from unknown questions.
New methods for explaining Random Forest predictions using case-based reasoning.
problem Lack of explainability for black-box machine learning models like Random Forests.
method Extracting distance metric from Random Forests to identify prototypes, critics, counter-factuals, and semi-factuals.
result Identified special points from training datasets to explain Random Forest predictions.
Method guarantees coherent factuality for language model outputs in reasoning tasks.
problem Ensuring correctness of language model outputs in reasoning tasks.
method Developed a conformal-prediction-based method applied to subgraphs within a deducibility graph.
result Achieved coherent factuality across target coverage levels, 90% on stricter definition.
Transformers can store facts efficiently using associative memories.
problem Understanding how transformers store and recall factual information.
method Proved linear scaling of storage capacities for linear and MLP associative memories, introduced a synthetic task, and analyzed gradient flow.
result Shallow transformers can achieve near optimal storage capacity for factual recall tasks using associative memories.
SLED improves factuality in LLMs without external knowledge.
problem Unreliable or factually incorrect outputs from large language models.
method Contrasts final layer logits with early layers' logits, uses approximate gradient to refine outputs.
result Consistently improves factual accuracy over existing methods.
MACI improves LLM factuality inference with higher retention and lower time cost.
problem Ensuring factuality in LLM responses for high-stakes domains.
method Reformulated conformal inference in a multiplicative filtering setting, leveraging ensembles for more accurate factuality scores and group-conditional calibration.
result MACI achieves higher retention and lower time cost compared to baselines, preserving validity through group-conditional calibration.
Performing inference on data obtained through observational studies is becoming extremely relevant due to the widespread availability of data in fields such as healthcare, education, retail, etc. Furthermore, this data is accrued from multiple homogeneous subgroups of a heterogeneous population, and hence, generalizing…
Proposes a deep learning framework for estimating counterfactual outcomes.
problem Challenges in estimating individual outcomes under different treatments.
method Deep variational Bayesian framework integrating factual and similar subjects' outcomes.
result Rigorously integrates individual features and similar subjects' responses for counterfactual outcomes.
MGM improves media profiling by integrating textual and structural features.
problem Evaluating the bias and factuality of news media from digital data.
method MediaGraphMind (MGM) framework using variational EM, integrating textual and structural features.
result Achieves new state-of-the-art results in media profiling.
PO-Flow models potential and counterfactual outcomes for personalized treatment decisions.
problem Predicting individualized treatment effects from observational data.
method Continuous normalizing flow (CNF) framework for causal inference.
result Unified approach to potential outcome prediction, treatment effect estimation, and counterfactual prediction.
New findings show tool-augmented models can recall unlimited facts, outperforming purely memorized models.
problem Limitations of purely memorized models in recalling large amounts of factual information.
method Demonstrated the benefits of in-tool learning (external retrieval) over in-weight learning (memorization) for factual recall.
result Proved that tool-use enables unbounded factual recall via a simple and efficient circuit construction.
Graph signal processing detects hallucinations in large language models.
problem Detecting factual reasoning from hallucinations in large language models.
method Modeling transformer layers as dynamic graphs, using spectral analysis to define diagnostics.
result Spectral signatures can distinguish different types of hallucinations and achieve high accuracy.
Bounds on factual and counterfactual distributions under measurement error in discrete models.
problem Measurement errors in discrete data and their impact on inference.
method Expressing modeling assumptions as linear constraints and using linear programming to derive bounds.
result Sharp bounds on factual and counterfactual distributions for various models, including instrumental variable scenarios.
We present SemEval-2019 Task 8 on Fact Checking in Community Question Answering Forums, which features two subtasks. Subtask A is about deciding whether a question asks for factual information vs. an opinion/advice vs. just socializing. Subtask B asks to predict whether an answer to a factual question is true, false or…
Estimates counterfactual outcomes linking observed and unobserved data.
problem Estimating expected counterfactual outcomes for individuals.
method Introduces retrospective counterfactual estimators and prediction intervals linking observed and unobserved outcomes.
result Retrospective counterfactual estimators and prediction intervals asymptotically satisfy valid coverage under standard causal assumptions.
This study quantifies uncertainty in comparing treatments using RCTs with before-and-after measures.
problem Uncertainty in comparing treatments using RCTs with before-and-after measures.
method New statistical modeling principle called ETZ enables counterfactual uncertainty quantification (CUQ) in RCTs with Before-and-After Repeated Measures.
result CUQ typically has lower variability than factual uncertainty quantification and can be achieved in RCTs.
We present a study on predicting the factuality of reporting and bias of news media. While previous work has focused on studying the veracity of claims or documents, here we are interested in characterizing entire news media. These are under-studied but arguably important research problems, both in their own right and …
CausalLongPFN predicts counterfactual outcomes from time-series data.
problem Predicting future outcomes under varying treatments in time-series data with confounding and heterogeneity.
method Prior-fitted network pretrained on synthetic episodes of temporal structural causal models.
result CausalLongPFN outperforms domain-trained models on factual and counterfactual prediction tasks.
Benchmark evaluates AI-generated financial QA hallucinations, highlighting system vulnerabilities.
problem Ensuring factual accuracy of AI-generated financial QA outputs.
method Developed a benchmark dataset and evaluated six detection methods under clean and noisy conditions.
result LLM-based judges and embedding methods perform best, but degrade under noisy conditions.
Knowledge bases (KBs) are the backbone of many ubiquitous applications and are thus required to exhibit high precision. However, for KBs that store subjective attributes of entities, e.g., whether a movie is "kid friendly", simply estimating precision is complicated by the inherent ambiguity in measuring subjective phe…
Paper analyzes biases in video QA datasets, showing models can answer 37-48% questions correctly without multimodal context.
problem Question answering biases in video QA datasets can lead to model overfitting and poor generalization.
method Analyzed popular video question answering datasets, conducted ablation studies on biases from annotators and question types.
result Pretrained language models can answer 37-48% questions correctly without multimodal context, far exceeding random guess baseline.
Machine learning improves learning and memory retention by optimizing study sessions.
problem Improving learning and memory retention methods for factual material.
method Large-scale randomized controlled trial with machine learning optimization of study sessions.
result Study sessions optimized with machine learning lead to 67% longer retention and 50% higher return rate.
Selection of input features such as relevant pieces of text has become a common technique of highlighting how complex neural predictors operate. The selection can be optimized post-hoc for trained models or incorporated directly into the method itself (self-explaining). However, an overall selection does not properly c…
Causal inference analysis is the estimation of the effects of actions on outcomes. In the context of healthcare data this means estimating the outcome of counter-factual treatments (i.e. including treatments that were not observed) on a patient's outcome. Compared to classic machine learning methods, evaluation and val…
Sharp limits found for storing and retrieving input-output associations in linear associative memories.
problem Understanding the fundamental limits of storing and retrieving input-output associations in neural networks.
method Study of a minimal linear associative memory model, introducing a decoupled model and using statistical physics to characterize storage capacity.
result Linear associative memory can store up to 1/2 log(p) associations, providing a sharp statistical-physics characterization.
Prognosticator improves performance in non-stationary MDPs.
problem Real-world applications often violate the stationary assumption in reinforcement learning.
method A policy gradient algorithm that maximizes a forecast of future performance.
result Prognosticator outperforms online adaptation techniques in non-stationary MDPs.
CVIB uses information theory to learn counterfactuals from MNAR data without RCTs.
problem Debiasing learning from missing-not-at-random (MNAR) data in recommendation systems.
method CVIB, a variational information bottleneck, separates task-aware mutual information into factual and counterfactual parts.
result CVIB significantly enhances both shallow and deep models in recommendation systems.
We present FAKTA which is a unified framework that integrates various components of a fact checking process: document retrieval from media sources with various types of reliability, stance detection of documents with respect to given claims, evidence extraction, and linguistic analysis. FAKTA predicts the factuality of…
Framework controls uncertainty in LLMs without labels or probabilities.
problem Managing uncertainty in black-box LLMs without token-level probability or true labels.
method Integrates generative models, UCP, and conformal alignment to control uncertainty.
result Achieves close-to-nominal coverage and tighter thresholds than split UCP.
DS-CP improves reliability of uncertainty quantification for large language models under domain shift.
problem Overconfident and factually incorrect outputs (hallucinations) from large language models.
method Adapts conformal prediction to large language models under domain shift by reweighting calibration samples.
result DS-CP delivers more reliable coverage than standard conformal prediction, especially under substantial distribution shifts.
This paper introduces DCE for better counterfactual explanations using optimal transport.
problem Lack of nuanced distributional characteristics in existing counterfactual explanations.
method Formulates a chance-constrained optimization problem using optimal transport to derive counterfactual distributions.
result DCE provides deeper insights into decision-making models by aligning counterfactual distributions with factual ones.
HabitatAgent offers a multi-agent system for transparent housing consultation.
problem Opaque reasoning and brittle multi-constraint handling in housing recommendation systems.
method HabitatAgent is a multi-agent architecture with specialized roles for memory, retrieval, generation, and validation.
result HabitatAgent achieves 95% accuracy in real user consultation scenarios, significantly outperforming a strong baseline.
LLMs compress financial texts, but distort decision-making.
problem LLMs compress financial texts, altering decision-making.
method Analyzed two diagnostic patterns: decontextualization and model dependency. Proposed Agentic Context Compression.
result LLM-compressed financial texts alter decision-making.
Transformers can be hijacked by context, but deeper models are more robust.
problem Robustness of Transformers against context hijacking for linear classification.
method Developed a theoretical analysis on the robustness of linear transformers, considering model depth, training context lengths, and number of hijacking context tokens.
result Deeper transformers are more robust to context hijacking.
A brief historical perspective is first given concerning financial crashes, - from the 17th till the 20th century. In modern times, it seems that log periodic oscillations are found before crashes in several financial indices. The same is found in sand pile avalanches on Sierpinski gaskets. A discussion pertains to the…
Language models learn automotive complaints, improving defect detection.
problem Detecting technical quality issues in customer feedback.
method Continual pre-training on ODI Complaints data set.
result Models achieve high precision in identifying technical terms.
Tractograms are mathematical representations of the main paths of axons within the white matter of the brain, from diffusion MRI data. Such representations are in the form of polylines, called streamlines, and one streamline approximates the common path of tens of thousands of axons. The analysis of tractograms is a ta…
Cryptos remained resilient after SVB's collapse, contrary to expectations.
problem Impact of SVB collapse on crypto markets.
method Factual summary, sentiment analysis, and market performance examination.
result Cryptocurrencies showed resilience after SVB's collapse.
Paper introduces SDM for detecting LLM hallucinations, improving on entropy tests.
problem Challenges of Large Language Models (LLMs) with non-factual, nonsensical responses.
method Joint clustering on sentence embeddings to measure semantic divergence between prompts and responses.
result SDM framework detects deeper form of arbitrariness in LLM responses.
Algorithm simulates counterfactuals for fairness analysis.
problem Analytical intractability of counterfactuals in conditional distributions.
method Proposes an algorithm using particle filtering for discrete and continuous variables.
result Asymptotically valid inference for counterfactuals.
The article describes our submission to SemEval 2019 Task 8 on Fact-Checking in Community Forums. The systems under discussion participated in Subtask A: decide whether a question asks for factual information, opinion/advice or is just socializing. Our primary submission was ranked as the second one among all participa…
We investigate the problem of estimating the causal effect of a treatment on individual subjects from observational data, this is a central problem in various application domains, including healthcare, social sciences, and online advertising. Within the Neyman Rubin potential outcomes model, we use the Kullback Leibler…
Generates counterfactuals in target domain from source domain observations.
problem Cross-domain learning with domain shifts and lack of parallel datasets.
method Unsupervised, Neural Causal Models, Joint Causal Graphs, Effect-Intrinsic vs Domain-Intrinsic Variables.
result Framework generates counterfactuals that closely match ground truth.
Study evaluates five LLMs for financial report analysis, revealing performance differences and variability.
problem Lack of understanding in reliability, consistency, and transparency of LLMs in financial analysis.
method Human evaluation, automated similarity metrics, and behavioral diagnostics applied to five transformer-based LLMs over U.S. 10-K filings.
result No single LLM consistently dominates across all evaluation perspectives, highlighting variability and need for interpretability.
In this paper, we consider a generic probabilistic discriminative learner from the functional viewpoint and argue that, to make it learn well, it is necessary to constrain its hypothesis space to a set of non-trivial piecewise constant functions. To achieve this goal, we present a scalable unsupervised regularization f…
We contribute the largest publicly available dataset of naturally occurring factual claims for the purpose of automatic claim verification. It is collected from 26 fact checking websites in English, paired with textual sources and rich metadata, and labelled for veracity by human expert journalists. We present an in-de…
SurvCaus improves survival CATE estimation using neural nets.
problem Estimating Individual Treatment Effects (ITE) in survival analysis.
method Representation balancing for counterfactual inference with neural networks.
result The proposed method outperforms baseline methods in synthetic and semisynthetic datasets.
Paper develops a model for verifying facts in tables without pre-retrieved evidence.
problem Verification of factual claims in structured data, especially in open-domain settings.
method Joint reranking-and-verification model that fuses evidence documents.
result Model achieves comparable performance to closed-domain state-of-the-art on TabFact dataset.