Neural networks help auditors efficiently assess financial statements by learning underlying data patterns.
problem Efficiently auditing large volumes of financial statements and journal entries.
method Vector Quantised-Variational Autoencoder (VQ-VAE) neural networks.
result VQ-VAE neural networks can learn a quantized representation of accounting data, uncovering latent factors and providing a representative audit sample.
This study improves audit sampling by using sequential procedures with statistical guarantees.
problem Improving audit efficiency and reliability with statistical methods.
method Formulated as a sequential testing problem, defining null and alternative hypotheses, stopping and decision rules, and exact boundary conditions.
result Exact design yields ex ante control of decision error probabilities, and simulation-based implementation approximates this design.
Develops Active Fourier Auditor to estimate ML model properties without reconstructing them.
problem Verifying and auditing properties of Machine Learning models in real-world applications.
method A new framework that quantifies ML model properties using Fourier coefficients, without reconstructing the model.
result Active Fourier Auditor (AFA) is more accurate and sample-efficient than baselines for estimating robustness, individual fairness, and group fairness.
Study examines auditing fairness in evolving models, identifying strategic updates that preserve audit properties.
problem Auditing fairness in machine learning models that adapt to changing environments.
method Characterizes strategic updates that preserve audit properties, proposes a generic PAC auditing framework.
result Establishes distribution-free auditing bounds for statistical parity using the SP dimension.
New divergences help audit DP in high dimensions.
problem Challenges in auditing DP in high-dimensional data.
method Propose kernel Rényi divergence and its regularized version for auditing.
result Regularized kernel Rényi divergence can be estimated from samples in high dimensions.
RLFA estimates misstated monetary fraction with weighted sampling without replacement.
problem Estimating misstated monetary fraction with given accuracy and confidence.
method Developed new confidence sequences for weighted average of unknown values using randomized weighted sampling and side information.
result Adaptive methods improve accuracy of estimates based on side information's predictive power.
Auditing fairness of decision-makers is now in high demand. To respond to this social demand, several fairness auditing tools have been developed. The focus of this study is to raise an awareness of the risk of malicious decision-makers who fake fairness by abusing the auditing tools and thereby deceiving the social co…
New method audits DP guarantees without noise or subsampling info.
problem Auditing DP guarantees of ML models without prior info.
method Histogram-based density estimation for lower bounds.
result Natural generalization of membership inference auditing.
Efficiently audits model fairness with continuous monitoring and flexible data collection.
problem Continuous monitoring and flexible data collection for fairness auditing.
method Sequential, anytime-valid inference and game-theoretic statistics.
result Demonstrated efficacy on three fairness datasets.
Proposes auditing for envy-freeness in recommender systems to assess individual preferences.
problem Auditing fairness in recommender systems for individual preferences.
method Formulates a pure exploration problem in multi-armed bandits, proposing a sample-efficient algorithm with theoretical guarantees.
result Algorithm ensures fairness without deteriorating user experience on real-world datasets.
Proposes PA-DSL for correcting noisy human labels in automated data labeling.
problem Noisy human labels in automated data labeling.
method Uses adjudicated cases to correct noisy human labels and debias analyses.
result Maintains nominal coverage and reduces RMSE by 10-17% relative to using only adjudicated labels.
Fine-tuning LLMs on privacy-sensitive data introduces privacy risk, and synthetic data audits can quantify this risk.
problem Fine-tuning LLMs on privacy-sensitive data introduces privacy risk.
method Generate synthetic canaries via high-temperature sampling from LLMs.
result Synthetic canaries are high-influence outliers that ensure strong audits.
Algorithm identifies best arm with biased proxy and selective ground truth audits.
problem Fixed-confidence best-arm identification with biased proxy and selective ground truth.
method Propensity-weighted estimator and adaptive auditing algorithm.
result Plug-in Neyman rule achieves near-oracle audit efficiency.
Study cost-effective fairness audits with partial feedback, improving over random exploration.
problem Auditing fairness of classifiers with limited true labels.
method Introduces cost model, proposes near-optimal algorithms for black-box and mixture models.
result Significantly lower audit costs compared to natural baselines.
New auditors assess f-DP privacy with adaptive sampling, avoiding large sample sizes.
problem Empirical auditing of f-DP privacy with adaptive sampling. method Shift focus to f-DP, develop adaptive auditors for whitebox and blackbox settings. result Adaptive auditors detect f-DP violations across the privacy spectrum with statistical guarantees. The paper audits trading filters, finding a high save-to-miss ratio.
problem Improving the efficiency and accuracy of trading filters in decentralized exchanges.
method A precision audit of filter rules against real trading data, classifying rejection events.
result Conservative save-to-miss ratio of 3.7 : 1, with wider interpretation of 14.8 : 1.
Audit financial machine learning workflows to detect spurious predictability.
problem Spurious predictability in financial machine learning models.
method Falsification audit testing predictive workflows against synthetic environments.
result Many apparent financial predictions are artifacts, not genuine.
Community moderation drifts towards majority, study finds.
problem How to ensure crowd-sourced moderation systems trust and reward accurate evaluations.
method Consensus-based auditing with a two-stage algorithm that weights contributors by the stability of their past residuals.
result Minority contributors' evaluations drift towards the majority, and their participation share falls on controversial topics.
New method improves model risk prediction using cross-audit projection.
problem Over-optimism in K-fold CV for binary classification. method Cross-audit projection (CAP) procedure combining resampling and asymptotic bias correction.
result CAP estimator achieves second-order asymptotic unbiasedness.
CJE calibrates cheap LLM judges against an oracle, achieving high accuracy at a fraction of the cost.
problem Inexpensive LLM judges can produce biased rankings, leading to unreliable outcomes.
method CJE uses a small oracle to calibrate cheap scores, then evaluates at scale with valid uncertainty.
result CJE achieves 99% pairwise ranking accuracy at 14x lower cost compared to a 16x oracle/judge cost ratio.
Adaptive auditing improves AI robustness testing with anytime-valid guarantees.
problem Cost and time of annotation limit rigorous AI failure mode characterization.
method Introduces hypothesis testing framework for adaptive audits using SAVI.
result Proves anytime-valid type-I error control and robustness certification.
Signed compression progress on a sealed audit is goodhart-resistant.
problem Intrinsic motivation for agents to improve their world models by compressing experience.
method Rewarding agents for the signed decrease of a fixed sealed-audit loss.
result Cumulative reward telescopes exactly to endpoint audit improvement, preventing infinite reward push while true audit performance stagnates.
Mathematical model audits social media algorithms to prevent bias.
problem Algorithmic filtering can bias users' decisions and societal norms.
method Formalized mathematical framework for auditing social media algorithms.
result Data-driven statistical auditing procedure to regulate algorithmic bias.
Big data transforms accounting and auditing, enhancing insights but posing challenges.
problem Challenges in data privacy and security with increased data sources.
method Utilizing AI and machine learning for efficient data analysis and anomaly detection.
result Enhanced analytics tools and continuous learning are key to overcoming challenges.
AI task delegation faces incentive collapse with unbounded payments as AI accuracy rises.
problem Incentive collapse in AI-assisted task delegation schemes.
method General impossibility result and sentinel-auditing payment mechanism.
result Sentinel-auditing mechanism enforces positive human effort at finite cost, independent of AI accuracy.
RESHAPE explains financial statement anomalies by aggregating explanations from AENNs.
problem Detecting and explaining accounting anomalies in financial audits is challenging.
method Proposes RESHAPE to explain model output on an aggregated attribute-level.
result RESHAPE provides more comprehensible explanations compared to existing methods.
AI-driven framework improves enterprise financial audits and risk identification.
problem Manual auditing is inefficient and limited by data complexity and evolving fraud tactics.
method Machine learning algorithms (SVM, RF, KNN) applied to a dataset of audit project counts, violations, and fraud instances.
result Random Forest achieves best performance with F1-score of 0.9012, identifying fraud and compliance anomalies.
Improved canary crafting for one-run privacy auditing reduces leakage estimates.
problem Detecting canaries in one-run privacy auditing to estimate leakage effectively.
method Optimizes canaries for detectability and diversity, using a greedy initialization and bilevel optimization.
result Achieves stronger leakage estimates at lower computational cost.
Fairness audits fail under missing protected labels, especially at zero access.
problem Understanding the reliability of fairness audits with incomplete protected-label data.
method Introduced a seed-calibrated stress test to separate missingness effects from seed-to-seed movement.
result Missing protected labels do not significantly alter fairness mitigation methods, but they can lead to harmful intersectional outcomes.
Audited Conformal Prediction improves conditional coverage in pretrained models under distribution shift.
problem Uncertainty quantification for pretrained models under unknown distribution shift
method Leverages a small labeled dataset to train an audit model for marginal coverage, integrates outputs into conformal prediction framework
result Significantly higher conditional coverage than existing approaches
To help enforce data-protection regulations such as GDPR and detect unauthorized uses of personal data, we develop a new \emph{model auditing} technique that helps users check if their data was used to train a machine learning model. We focus on auditing deep-learning models that generate natural-language text, includi…
Study efficient auditing of ML fairness models.
problem Scalability of auditing ML models for fairness.
method Query-based auditing algorithms for estimating demographic parity.
result Optimal deterministic and practical randomized algorithms for fairness estimation.
JAWS audits predictive uncertainty under covariate shift using jackknife+ weighted methods.
problem Auditing predictive uncertainty under data distribution shifts.
method JAW and JAWA methods for distribution-free uncertainty quantification.
result JAW relaxes the jackknife+'s assumption of data exchangeability for covariate shift.
Develops tools to audit ML models for bias and unfairness.
problem Auditing ML models for individual bias and unfairness.
method Formalizes the task as an optimization problem and develops inferential tools for the optimal value.
result Demonstrates the utility of tools in revealing biases in COMPAS recidivism prediction instrument.
Survey of determinism issues in financial AI systems.
problem Vulnerabilities in reproducibility of financial AI systems.
method Literature review and first-party experiments on public financial datasets.
result Proposed a layered evaluation framework linking modality-specific metrics to audit readiness.
Motivated by the need to audit complex and black box models, there has been extensive research on quantifying how data features influence model predictions. Feature influence can be direct (a direct influence on model outcomes) and indirect (model outcomes are influenced via proxy features). Feature influence can also …
The rapid growth of text data has motivated the development of machine-learning based automatic text summarization strategies that concisely capture the essential ideas in a larger text. This study aimed to devise an extractive summarization method for A-133 Single Audits, which assess if recipients of federal grants a…
The study examines robustness auditing for linear regression, improving existing methods and identifying computational challenges.
problem Detecting small subsets of data that can reverse regression coefficients.
method Empirical study of mixed integer quadratically constrained optimization and exact greedy methods, combined with a spectral algorithm.
result Existing methods largely outperform state of the art, but computational bottlenecks remain, especially for higher dimensions.
New method protects whistleblowers from retaliation by ensuring their reports remain private.
problem Whistleblowers face retaliation, and current protections are insufficient.
method Formalizes protection against strong-adversary threat model as per-report (0,δ)-differential privacy, and provides a generic mechanism to reduce private auditing to private continual counting. result Demonstrates a reduction in selection error and improved utility over randomized response.
Study reveals AI skin cancer classifiers underperform for darker skin phototypes, advocating for fairness auditing.
problem AI bias in dermatology, particularly for darker skin phototypes.
method Predictive Representativity (PR) framework, evaluating classifiers on HAM10000 and BOSQUE Test sets.
result Substantial performance disparities by skin phototype, highlighting AI bias.
ISAAC audits deep models for drug-target interactions, revealing structural differences.
problem Deep models for DTI often use irrelevant features, making them hard to evaluate.
method ISAAC uses intervention-based structural auditing to evaluate model sensitivity.
result ISAAC reveals significant structural differences in DTI models' reasoning.
Audit shows risk claims from distributional reinforcement learning agents are often false.
problem Evaluating the risk claims made by distributional reinforcement learning agents.
method Combines a decision-relevant screening metric, ground truth from Monte Carlo, and statistical methods to audit risk claims.
result 40-95% of the strongest risk claims are refuted, indicating the learned risk reflects a training artifact rather than environment stochasticity.
Peer-induced fairness framework audits algorithmic fairness in AI applications.
problem Current auditing methods lack robustness and fail to distinguish between algorithmic discrimination and subject limitations.
method Combines counterfactual fairness and peer comparison strategy for a reliable auditing tool.
result Demonstrates significant unfairness in micro-firms compared to non-micro firms, highlighting the framework's potential.
New metric evaluates generative models across domains, diagnosing fidelity, diversity, and generalization.
problem Evaluating generative models in diverse domains with limited metrics.
method Introduces a 3D evaluation metric (α-Precision, β-Recall, Authenticity) for domain-agnostic diagnostics. result Unified metric characterizes fidelity, diversity, and generalization, diagnosing model performance.
Paper tackles transparency and auditability of machine learning in credit scoring.
problem Missed potential in using modern machine learning for credit scoring due to lack of transparency.
method Develops a framework for making black box machine learning models transparent, auditable, and explainable.
result Comparable interpretability can be achieved with machine learning while maintaining predictive power.
Framework uses hindsight regret to audit marketing budget allocations.
problem Lack of principled way to assess strategic budget allocations.
method Hindsight regret framework based on constraint-faithful benchmark.
result Identifies practical trade-off between allocation flexibility and detectability.
Framework audits synthetic datasets for trustworthiness across various use cases.
problem Assessing the trustworthiness of synthetic datasets and models.
method Holistic auditing framework focusing on bias, fidelity, utility, robustness, and privacy.
result Introduces a trustworthiness index and model selection process for controllable trade-offs.
AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing
problem Auditing LLM-as-a-Judge decisions
method Adaptive uncertainty-aware refinement
result Human-consistent signal learning and evidence propagation