AUDITA: certified auditing and causal attribution of adverse outcomes in autonomous multi-agent systems

Extended abstract

This paper presents Audita, an audit layer that grades each agent's responsibility for an adverse outcome in multi-agent systems such as LLM-commanded robot fleets, where existing methods name a single culprit from unverifiable logs. Each inter-agent message is an Ed25519-signed tuple of author, payload, citations, and committed effect, sealed into append-only Merkle-committed epochs. A production gate retires commands whose effect never reached the outcome; survivors are graded by the modified Halpern-Pearl definition with the Chockler-Halpern degree rho = 1/(|X*|+|W*|), covering overdetermination, preemption, and omissions as duty-indexed absence variables. Each cause is certified by counterfactual replay, bitwise exact on pinned open-weight models in vLLM and Hoeffding-bounded on hosted ones. The output is a per-agent negligence-law duty, breach, causation, and harm verdict, matching the tort-law NESS test on monotone models.

Under replayable models, a recorded channel, and unstolen keys, the paper proves that forgeries cannot change the verdict and deletions only widen a responsibility interval; that a duty-compliant agent framed by malicious insiders into a true but-for cause is never found culpable, while a replay-checkable certificate grades the framers; and that when agents make honest errors, no record-based breach standard is both sound and complete.

On live GSM8K planner-solver-aggregator-verifier pipelines with Qwen2.5-72B-Instruct, Audita cuts the Who&When LLM judge's responsibility error from 0.589 to 0.170 with the answer key and 0.159 without it, against 0.223 for Shapley attribution and 0.408 for both CAR/CausalFlow single-site scoring and duty-ablated Audita. 95-98% of failures are joint solver-aggregator causes, 58.7% of Who&When traces show a duty-bearing verifier or orchestrator staying silent, and changing only the duty standard flips culpability in 69.8% of cases. Shapley wins on Qwen2.5-7B (0.041 versus 0.111) but trails on Llama-3.1-8B and Qwen2.5-72B. On Who&When's 184 static traces, a variant without replay reaches agent accuracy 0.408 and step accuracy 0.321, matching LLM judges.

The paper releases CulpaBench, over 1,000 multi-robot facility incidents grounded in 42 OSHA/NIOSH accident records, with independent-oracle ground truth. Audita is exact (error 0.000) on redundancy, a transcript-identical inert twin, preemption, omission, and delegation chains, against averages of 0.303-0.343 for LLM judges, 0.433 single-site, 0.452 Shapley, and 0.900 for an anomaly detector. On 300 no-cause accidents, judges blame someone in 100% of runs even when offered NONE; Audita never does. On 74 held-out omitted-halt incidents, judges convict an innocent in 67-100% of runs. Forgeries push an unsigned log to 0.365 error, framing an innocent in all 450 probes, while the certified verdict stays exact; under 600 insider attacks victim culpability stays 0.000, and a level-2 scapegoat succeeds in 0 of 330 draws.

Exact computation rises from 0.02 s at 2 candidates to 62.1 s at 14, at 393 B per message. Moving 31% of coordination off the record hides 0.157 of responsibility with zero over-attribution; grades resist re-chunking and log flooding, and verdict-guided repair averts 2.4x the harm of a random fix. Evidence is simulated and limited to discrete threshold incidents; the paper argues EU AI Act records must be signed, cited, sealed, and causally analysable.

Questions this paper answers (18)

How can I attribute blame to multiple agents when an LLM multi-agent pipeline fails, instead of naming one culprit?

If you work on automated failure attribution for LLM multi-agent pipelines, this paper replaces single-culprit labels with a per-agent graded responsibility verdict and, on live GSM8K planner-solver-aggregator-verifier pipelines, cuts the Who&When-style LLM judge's responsibility error from 0.589 to 0.170 (gold-informed) and 0.159 (no answer key) (Results, Table 1).

Topic: Graded failure attribution for LLM multi-agent systems. Evidence: Results: Attribution on live multi-agent failures; Table 1. Also asked as: Which methods improve on LLM-as-judge failure attribution for multi-agent systems?

Are failures in multi-agent LLM pipelines usually caused by one agent or by several agents jointly?

If you study why multi-agent LLM systems fail, this paper finds that under duty-aware graded ground truth 95-98% of adverse incidents (56/57, 43/47, 39/41) are joint solver-aggregator incidents, none fit the single-culprit structure, and in 58.7% of Who&When traces a duty-bearing verifier or orchestrator stayed silent through the decisive error (Results, live failures).

Topic: Most multi-agent failures are jointly caused. Evidence: Results: Attribution on live multi-agent failures. Also asked as: Does the single-culprit assumption in multi-agent failure attribution benchmarks hold?

How do I compute Chockler-Halpern degree of responsibility for agents in a multi-agent system?

If you apply actual causality or degree-of-responsibility theory to AI systems, this paper builds a structural causal model over certified inter-agent messages and duty-indexed absence variables, grades causes with the modified Halpern-Pearl definition and the Chockler-Halpern degree rho = 1/(|X*|+|W*|), and handles overdetermination (1/2, 1/2 split), preemption and omission natively (Methods, Eqs. 1-2).

Topic: Halpern-Pearl actual causality for agent messages. Evidence: Methods: Certified causal attribution; Eqs. (1)-(2); Supplementary Note S1. Also asked as: Applying the modified Halpern-Pearl definition of actual causation to LLM agent logs

How to handle preemption when doing counterfactual blame attribution over logs?

If you need to separate causal production from counterfactual dependence, this paper adds a gate that retires commands whose committed effect never propagated to the outcome before any counterfactual is computed (Proposition 2), and shows the gate is load-bearing on preemption and on a transcript-identical 'inert twin' that every transcript-level method misclassifies (Ablations; Extended Data Fig. S1).

Topic: Production gate before counterfactual grading. Evidence: Methods (Gate); Proposition 2; Results: CulpaBench, Ablations; Supplementary Note S5. Also asked as: Why can two identical transcripts require different causal attributions?

What should a tamper-evident audit log for LLM multi-agent systems contain?

If you design audit logs for AI agents, this paper specifies a message format (author, payload, citation set, committed effect predicate, Ed25519 signature) sealed into Merkle-committed append-only epochs, proves forgeries leave the verdict unchanged and deletions only widen a responsibility interval (Proposition 3), and measures that forgeries drive an unsigned log to 0.365 error and frame an innocent in all 450 probes while the certified verdict stays at 0.000 (Figure 3b).

Topic: Tamper-evident signed logs for agent accountability. Evidence: Methods: accountable command graph, Definition 1, Proposition 3; Figure 3b; Extended Data Fig. 1; Supplementary Note S7. Also asked as: Can log forgery or deletion change blame attribution for AI agents, and how to prevent it?

Can a malicious agent frame an innocent agent in a causal responsibility analysis?

If you study adversarial manipulation of blame or responsibility, this paper constructs two conduct-level attacks (conditioning and inevitability rigging) that make a compliant agent a true but-for cause, proves such an agent is never found culpable (Theorem 1) and that a replay-checkable blame-shift certificate grades the framers (Theorem 2, extended to all orders in Theorem 5); over 600 incidents victim involvement reaches 1.0 but culpability stays 0.000, and a level-2 scapegoat convicts a benign deviator in 0 of 330 draws (Results, Robustness).

Topic: Framing attacks by insiders with valid keys. Evidence: Methods: Guarantees; Theorems 1, 2, 5; Proposition 4; Supplementary Notes S3-S4; Results: Conduct attacks. Also asked as: Non-frameability guarantees for blame attribution in multi-agent systems

Is it possible to audit AI agent failures soundly and completely without ground-truth labels?

If you ask what an auditor can certify without an answer key, this paper proves a completeness barrier: when compliant agents can make honest errors, no breach standard computed from the record is both sound and complete, and closing the gap amounts to verifying task correctness (Theorem 4); empirically the gold-free deployable verdict matches the gold-informed one (0.159 vs 0.170) (Ablations; Remark S8).

Topic: Limits of ground-truth-free auditing. Evidence: Theorem 4; Supplementary Note S3 (Theorem S3, Remark S8); Results: Ablations. Also asked as: Theoretical limits of blame attribution when agents make honest mistakes

How can tort law concepts like duty, breach and NESS causation be formalized for autonomous agents?

If you connect legal causation to AI accountability, this paper outputs a per-agent duty-breach-causation-harm verdict from negligence law that can report a negligent-but-inert agent or an involved-but-not-culpable conduit, and proves that on monotone incident models its causes coincide with the NESS test from tort doctrine via minimal transversals of prime implicants (Remark 1; Proposition S1; Figure 2).

Topic: Negligence doctrine and NESS test for AI blame. Evidence: Methods: From causes to verdicts; Remark 1; Figure 2; Supplementary Note S2 (Proposition S1). Also asked as: Relationship between the Halpern-Pearl definition and the NESS test

Do LLM judges over-attribute blame when no agent is at fault?

If you evaluate LLM-as-a-judge for incident analysis, this paper shows that on 300 no-cause accidents LLM judges named a culprit in 100% of runs across all three Who&When formats, even with a prompt offering NONE, and an anomaly detector did so in every case, while the certified engine manufactured a culprit in zero cases (Results, Negative controls).

Topic: LLM judges blame someone even with no culprit. Evidence: Results: CulpaBench, Negative controls; Table 2. Also asked as: Negative controls for automated failure attribution methods

Is there a benchmark for causal responsibility attribution with planted ground truth?

If you need a benchmark with analytic ground truth for responsibility attribution, this paper releases CulpaBench, over 1,000 incidents in a deterministic multi-robot facility whose structures (redundancy, inert twin, preemption, omission, delegation chain, record attack, no-cause and trivial controls) are grounded in 42 public OSHA/NIOSH robot-accident records, with ground truth from an independent oracle sharing no code with the engine (Table 2; Extended Data Table 1; Note S5).

Topic: CulpaBench benchmark from robot accident records. Evidence: Results: CulpaBench; Table 2; Extended Data Table 1; Supplementary Notes S5, S5.1. Also asked as: Datasets of robot accident structures for evaluating blame attribution

How to attribute responsibility to an agent that failed to act, such as a missing safety halt?

If you work on blaming inaction, this paper encodes required-but-unissued commands as first-class absence variables and, on 74 held-out unexpected-startup incidents (the modal fatal pattern in US robot fatality data) where the only breach is an omitted halt, is exact while single-site counterfactual scoring blames the innocent sole actor every time and LLM judges convict an innocent in 67-100% of runs (Results, Generalisation).

Topic: Responsibility for omissions and unexpected startup. Evidence: Results: Generalisation; Extended Data Table 1; Supplementary Note S5. Also asked as: Causal attribution of omissions in multi-robot safety incidents

Is Shapley value attribution a good way to assign blame among LLM agents?

If you use Shapley values for multi-agent credit or blame, this paper finds Shapley the strongest baseline on live pipelines (0.223) but duty-blind on CulpaBench (average 0.452), and the comparison depends on model scale: Shapley wins on Qwen2.5-7B (0.041 vs 0.111) while the graded verdict leads on Llama-3.1-8B and Qwen2.5-72B, with margins -0.070, +0.037, +0.053 (Figure 3a; Table 2).

Topic: Shapley value versus causal responsibility for agents. Evidence: Table 1; Table 2; Results: Ablations; Figure 3a. Also asked as: How does failure attribution accuracy change across Qwen2.5-7B, Llama-3.1-8B and Qwen2.5-72B?

How to make LLM inference deterministic enough for counterfactual replay of agent runs?

If you need reproducible counterfactual replay of LLM agents, this paper certifies each reported cause by re-execution under a serial replay protocol on pinned open-weight models in vLLM (concurrent batching breaks bitwise exactness even in batch-invariant modes) and, for hosted non-deterministic models, gives a Hoeffding-based guarantee bounding false causes by alpha plus an action-match score (Proposition 1; Note S7).

Topic: Bitwise-exact LLM replay for counterfactual certification. Evidence: Methods: Certify (counterfactual replay); Proposition 1; Supplementary Notes S2, S7. Also asked as: Statistical guarantees for counterfactual causal claims with non-deterministic LLMs

Does the choice of normative standard change who is held responsible for an AI agent failure?

If you study normative standards in AI accountability, this paper shows that holding causal facts fixed and varying only the registered duty standard flips culpability in 69.8% of cases, and that removing the duty layer raises live error from 0.170 to 0.408, so a verdict must ship with its standard attached (Results, Ablations).

Topic: Verdicts depend on the declared duty standard. Evidence: Results: Ablations and analysis; Table 1; Table 2 (causal-only row). Also asked as: Separating causal involvement from culpability in AI incident analysis

What agent-level and step-level accuracy do methods reach on the Who&When failure attribution benchmark?

If you benchmark on Who&When, this paper reports a degraded structure-aware attributor without replay or cryptography reaching agent-accuracy 0.408 and step-accuracy 0.321 on the 184 static traces, within the Qwen2.5-72B judge band (0.41-0.51) and above two of three judge formats on step accuracy, arguing the advantage requires counterfactual access (Table 3).

Topic: Attribution on static logs: Who&When comparison. Evidence: Results: Comparability on the field benchmark; Table 3. Also asked as: Does counterfactual re-execution help multi-agent failure attribution compared with reading static logs?

How expensive is exact Halpern-Pearl causal computation on real incident graphs?

If you worry about the NP-hardness of actual causation in practice, this paper measures exact recovery at every tested incident width while wall-clock per query roughly doubles per added candidate (0.02 s at k=2 to 62.1 s at k=14), placing the practical limit near k=8 at a one-second budget, with record overhead of 393 B per signed message and 118 KB per incident (Extended Data Table 2; Costs and limits).

Topic: Computational cost of exact actual causation. Evidence: Results: Costs and limits; Extended Data Table 2; Extended Data Fig. 2. Also asked as: Storage overhead of cryptographically signed agent message logs

What happens to multi-agent blame attribution when agents coordinate through side channels not in the log?

If you ask how attribution behaves outside its assumptions or under manipulation, this paper moves 31% of coordination off the recorded bus and finds 0.157 of responsibility becomes invisible with over-attribution staying 0.000 over 1,200 draws, shows flooding the log with inert commands has slope 0.000 for all methods, proves grades are invariant to how an agent splits its output into messages (Proposition S3), and finds verdict-guided repair averts 2.4x the harm of a random fix (Costs and limits; Note S6).

Topic: Off-record influence, flooding and re-chunking robustness. Evidence: Results: Robustness (Volume manipulation), Costs and limits; Supplementary Note S6 (Proposition S3). Also asked as: Can an agent dilute its responsibility by splitting or flooding messages?

How should logging for high-risk AI systems under the EU AI Act be designed to support liability decisions?

If you work on incident investigation, liability or AI regulation for LLM-commanded robot fleets, this paper proposes what EU AI Act-mandated event records must be (signed, cited, sealed, causally analysable) to support accountability, positions itself relative to STAMP/CAST blame-free analysis, the ethical black box, PeerReview and Accountable Virtual Machines, and lists its scope limits: discrete threshold incidents only and simulated, not physical-robot, evidence (Discussion; Note S10).

Topic: Accident investigation and regulation for embodied AI fleets. Evidence: Introduction; Discussion; Supplementary Note S10. Also asked as: Accountability and incident forensics for multi-robot systems controlled by LLMs

@misc{du2026audita,
  title         = {AUDITA: certified auditing and causal attribution of adverse outcomes in autonomous multi-agent systems},
  author        = {Du, Zhixu and Chen, Yiran},
  year          = {2026},
  eprint        = {2608.22160},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2608.22160}
}

Page ID: zdu-html-a65ce9