Signed compression progress on a sealed audit is goodhart-resistant.
problem Intrinsic motivation for agents to improve their world models by compressing experience.
method Rewarding agents for the signed decrease of a fixed sealed-audit loss.
result Cumulative reward telescopes exactly to endpoint audit improvement, preventing infinite reward push while true audit performance stagnates.
This study improves audit sampling by using sequential procedures with statistical guarantees.
problem Improving audit efficiency and reliability with statistical methods.
method Formulated as a sequential testing problem, defining null and alternative hypotheses, stopping and decision rules, and exact boundary conditions.
result Exact design yields ex ante control of decision error probabilities, and simulation-based implementation approximates this design.
Privacy concerns have led to the development of privacy-preserving approaches for learning models from sensitive data. Yet, in practice, even models learned with privacy guarantees can inadvertently memorize unique training examples or leak sensitive features. To identify such privacy violations, existing model auditin…
Unified RMOT framework for non-modelable risk factors reduces audit bounds.
problem Infinite audit bounds for exotic derivatives pricing with sparse market data.
method Rough Martingale Optimal Transport (RMOT) with rough volatility regularization.
result Finite, explicit, and asymptotically tight extrapolation bounds for non-modelable risk factors.
Study examines auditing fairness in evolving models, identifying strategic updates that preserve audit properties.
problem Auditing fairness in machine learning models that adapt to changing environments.
method Characterizes strategic updates that preserve audit properties, proposes a generic PAC auditing framework.
result Establishes distribution-free auditing bounds for statistical parity using the SP dimension.
Framework detects and mitigates data-poisoning attacks in causal effect estimation.
problem Vulnerability to append-only attacks in observational causal analyses.
method Develops a data-poisoning audit for augmented inverse-probability-weighted estimation.
result Proposes a greedy scan to compute exact worst-case movement at every append budget.
Develops Active Fourier Auditor to estimate ML model properties without reconstructing them.
problem Verifying and auditing properties of Machine Learning models in real-world applications.
method A new framework that quantifies ML model properties using Fourier coefficients, without reconstructing the model.
result Active Fourier Auditor (AFA) is more accurate and sample-efficient than baselines for estimating robustness, individual fairness, and group fairness.
New method audits DP guarantees without noise or subsampling info.
problem Auditing DP guarantees of ML models without prior info.
method Histogram-based density estimation for lower bounds.
result Natural generalization of membership inference auditing.
Improved canary crafting for one-run privacy auditing reduces leakage estimates.
problem Detecting canaries in one-run privacy auditing to estimate leakage effectively.
method Optimizes canaries for detectability and diversity, using a greedy initialization and bilevel optimization.
result Achieves stronger leakage estimates at lower computational cost.
Neural networks help auditors efficiently assess financial statements by learning underlying data patterns.
problem Efficiently auditing large volumes of financial statements and journal entries.
method Vector Quantised-Variational Autoencoder (VQ-VAE) neural networks.
result VQ-VAE neural networks can learn a quantized representation of accounting data, uncovering latent factors and providing a representative audit sample.
The paper addresses fairness in online learning by extending auditing schemes and presenting efficient algorithms.
problem Ensuring fairness in online learning while maximizing predictive accuracy.
method Extending auditing schemes to handle multiple auditors and presenting oracle-efficient algorithms.
result Presented algorithms achieve upper bounds on regret and fairness violations, improving on existing bounds.
CJE calibrates cheap LLM judges against an oracle, achieving high accuracy at a fraction of the cost.
problem Inexpensive LLM judges can produce biased rankings, leading to unreliable outcomes.
method CJE uses a small oracle to calibrate cheap scores, then evaluates at scale with valid uncertainty.
result CJE achieves 99% pairwise ranking accuracy at 14x lower cost compared to a 16x oracle/judge cost ratio.
Algorithm reduces audit costs by identifying best service configurations from biased textual evidence.
problem Designing service systems from textual evidence requires accurate selection despite biased automated scoring.
method Developed PP-LUCB algorithm combining LLM scores and selective audits to minimize costs.
result Correctly identified the best model in 40/40 trials with 90% cost reduction.
A privacy-preserving synthetic data generation framework that distinguishes between true and phantom data disclosures.
problem Detecting and explaining data disclosures in synthetic datasets.
method Customizable empirical auditing framework with statistical hypothesis testing.
result Demonstrated tighter privacy leakage bounds than prior methods.
AI task delegation faces incentive collapse with unbounded payments as AI accuracy rises.
problem Incentive collapse in AI-assisted task delegation schemes.
method General impossibility result and sentinel-auditing payment mechanism.
result Sentinel-auditing mechanism enforces positive human effort at finite cost, independent of AI accuracy.
New method improves model risk prediction using cross-audit projection.
problem Over-optimism in K-fold CV for binary classification. method Cross-audit projection (CAP) procedure combining resampling and asymptotic bias correction.
result CAP estimator achieves second-order asymptotic unbiasedness.
Big data transforms accounting and auditing, enhancing insights but posing challenges.
problem Challenges in data privacy and security with increased data sources.
method Utilizing AI and machine learning for efficient data analysis and anomaly detection.
result Enhanced analytics tools and continuous learning are key to overcoming challenges.
RLFA estimates misstated monetary fraction with weighted sampling without replacement.
problem Estimating misstated monetary fraction with given accuracy and confidence.
method Developed new confidence sequences for weighted average of unknown values using randomized weighted sampling and side information.
result Adaptive methods improve accuracy of estimates based on side information's predictive power.
RESHAPE explains financial statement anomalies by aggregating explanations from AENNs.
problem Detecting and explaining accounting anomalies in financial audits is challenging.
method Proposes RESHAPE to explain model output on an aggregated attribute-level.
result RESHAPE provides more comprehensible explanations compared to existing methods.
AI-driven framework improves enterprise financial audits and risk identification.
problem Manual auditing is inefficient and limited by data complexity and evolving fraud tactics.
method Machine learning algorithms (SVM, RF, KNN) applied to a dataset of audit project counts, violations, and fraud instances.
result Random Forest achieves best performance with F1-score of 0.9012, identifying fraud and compliance anomalies.
New divergences help audit DP in high dimensions.
problem Challenges in auditing DP in high-dimensional data.
method Propose kernel Rényi divergence and its regularized version for auditing.
result Regularized kernel Rényi divergence can be estimated from samples in high dimensions.
Fine-tuning LLMs on privacy-sensitive data introduces privacy risk, and synthetic data audits can quantify this risk.
problem Fine-tuning LLMs on privacy-sensitive data introduces privacy risk.
method Generate synthetic canaries via high-temperature sampling from LLMs.
result Synthetic canaries are high-influence outliers that ensure strong audits.
Fairness audits fail under missing protected labels, especially at zero access.
problem Understanding the reliability of fairness audits with incomplete protected-label data.
method Introduced a seed-calibrated stress test to separate missingness effects from seed-to-seed movement.
result Missing protected labels do not significantly alter fairness mitigation methods, but they can lead to harmful intersectional outcomes.
Audited Conformal Prediction improves conditional coverage in pretrained models under distribution shift.
problem Uncertainty quantification for pretrained models under unknown distribution shift
method Leverages a small labeled dataset to train an audit model for marginal coverage, integrates outputs into conformal prediction framework
result Significantly higher conditional coverage than existing approaches
To help enforce data-protection regulations such as GDPR and detect unauthorized uses of personal data, we develop a new \emph{model auditing} technique that helps users check if their data was used to train a machine learning model. We focus on auditing deep-learning models that generate natural-language text, includi…
Study efficient auditing of ML fairness models.
problem Scalability of auditing ML models for fairness.
method Query-based auditing algorithms for estimating demographic parity.
result Optimal deterministic and practical randomized algorithms for fairness estimation.
Study cost-effective fairness audits with partial feedback, improving over random exploration.
problem Auditing fairness of classifiers with limited true labels.
method Introduces cost model, proposes near-optimal algorithms for black-box and mixture models.
result Significantly lower audit costs compared to natural baselines.
Develops tools to audit ML models for bias and unfairness.
problem Auditing ML models for individual bias and unfairness.
method Formalizes the task as an optimization problem and develops inferential tools for the optimal value.
result Demonstrates the utility of tools in revealing biases in COMPAS recidivism prediction instrument.
Algorithm identifies best arm with biased proxy and selective ground truth audits.
problem Fixed-confidence best-arm identification with biased proxy and selective ground truth.
method Propensity-weighted estimator and adaptive auditing algorithm.
result Plug-in Neyman rule achieves near-oracle audit efficiency.
Proposes PA-DSL for correcting noisy human labels in automated data labeling.
problem Noisy human labels in automated data labeling.
method Uses adjudicated cases to correct noisy human labels and debias analyses.
result Maintains nominal coverage and reduces RMSE by 10-17% relative to using only adjudicated labels.
Motivated by the need to audit complex and black box models, there has been extensive research on quantifying how data features influence model predictions. Feature influence can be direct (a direct influence on model outcomes) and indirect (model outcomes are influenced via proxy features). Feature influence can also …
The rapid growth of text data has motivated the development of machine-learning based automatic text summarization strategies that concisely capture the essential ideas in a larger text. This study aimed to devise an extractive summarization method for A-133 Single Audits, which assess if recipients of federal grants a…
Proposes auditing for envy-freeness in recommender systems to assess individual preferences.
problem Auditing fairness in recommender systems for individual preferences.
method Formulates a pure exploration problem in multi-armed bandits, proposing a sample-efficient algorithm with theoretical guarantees.
result Algorithm ensures fairness without deteriorating user experience on real-world datasets.
The paper shows how to audit fairness in decisions with hidden risk factors.
problem Estimating fairness in decisions influenced by hidden, unobservable risk factors.
method Derives unbiased estimates of risk using historical data and audits existing decision-making systems.
result One can compute meaningful bounds on treatment rates for high-risk individuals, even with hidden confounders.
New method protects whistleblowers from retaliation by ensuring their reports remain private.
problem Whistleblowers face retaliation, and current protections are insufficient.
method Formalizes protection against strong-adversary threat model as per-report (0,δ)-differential privacy, and provides a generic mechanism to reduce private auditing to private continual counting. result Demonstrates a reduction in selection error and improved utility over randomized response.
A new metric GNQ audits LLMs for privacy risks during training.
problem Auditing LLMs for privacy risks during training is computationally hard.
method Gradient Uniqueness (GNQ) metric derived from gradient descent, BS-Ghost GNQ for efficiency.
result GNQ successfully predicts sequence extractability and reveals risk heterogeneity.
Modeling incentives for content creators on algorithm-curated platforms.
problem Maximizing exposure for content creators on algorithmic platforms.
method Formalized exposure game model, proving effects of algorithmic choices on equilibria, proposing tools for finding equilibria.
result Algorithmic choices significantly affect content exposure and creator behavior.
ISAAC audits deep models for drug-target interactions, revealing structural differences.
problem Deep models for DTI often use irrelevant features, making them hard to evaluate.
method ISAAC uses intervention-based structural auditing to evaluate model sensitivity.
result ISAAC reveals significant structural differences in DTI models' reasoning.
Develops a method to continuously audit black-box conditional quantile forecasts.
problem Continuous monitoring of black-box forecasts under changing data streams and regimes.
method Distribution-free and game-theoretic testing framework for non-i.i.d. losses.
result Derives finite-time detection guarantees for miscalibrated forecasts based on features.
JAWS audits predictive uncertainty under covariate shift using jackknife+ weighted methods.
problem Auditing predictive uncertainty under data distribution shifts.
method JAW and JAWA methods for distribution-free uncertainty quantification.
result JAW relaxes the jackknife+'s assumption of data exchangeability for covariate shift.
Audit shows risk claims from distributional reinforcement learning agents are often false.
problem Evaluating the risk claims made by distributional reinforcement learning agents.
method Combines a decision-relevant screening metric, ground truth from Monte Carlo, and statistical methods to audit risk claims.
result 40-95% of the strongest risk claims are refuted, indicating the learned risk reflects a training artifact rather than environment stochasticity.
Peer-induced fairness framework audits algorithmic fairness in AI applications.
problem Current auditing methods lack robustness and fail to distinguish between algorithmic discrimination and subject limitations.
method Combines counterfactual fairness and peer comparison strategy for a reliable auditing tool.
result Demonstrates significant unfairness in micro-firms compared to non-micro firms, highlighting the framework's potential.
Paper tackles transparency and auditability of machine learning in credit scoring.
problem Missed potential in using modern machine learning for credit scoring due to lack of transparency.
method Develops a framework for making black box machine learning models transparent, auditable, and explainable.
result Comparable interpretability can be achieved with machine learning while maintaining predictive power.
Efficiently audits model fairness with continuous monitoring and flexible data collection.
problem Continuous monitoring and flexible data collection for fairness auditing.
method Sequential, anytime-valid inference and game-theoretic statistics.
result Demonstrated efficacy on three fairness datasets.
Framework uses hindsight regret to audit marketing budget allocations.
problem Lack of principled way to assess strategic budget allocations.
method Hindsight regret framework based on constraint-faithful benchmark.
result Identifies practical trade-off between allocation flexibility and detectability.
Firms miscount their customers who stop buying without saying goodbye.
problem Counting non-contractual customers accurately.
method Estimating repeat purchase probabilities and extrapolating to infinite time.
result The count of alive customers is only partially identified, with a wide range of estimates.
Framework audits synthetic datasets for trustworthiness across various use cases.
problem Assessing the trustworthiness of synthetic datasets and models.
method Holistic auditing framework focusing on bias, fidelity, utility, robustness, and privacy.
result Introduces a trustworthiness index and model selection process for controllable trade-offs.
Trimming helps in conformal prediction when it separates anomaly scores.
problem Effectiveness of trimming in conformal prediction under contamination.
method Analyse fixed-threshold trimming as a replacement of the contaminated calibration law with a retained law.
result Trimming helps when it separates anomaly scores, reducing clean-target coverage to a one-dimensional score-CDF transfer problem.