# Zhixu Du

> PhD candidate in Electrical and Computer Engineering at Duke University. I work on efficient machine learning systems (Mixture-of-Experts serving and load balancing, state space models), federated learning, and auditing of autonomous multi-agent AI systems.

Index ID: zdu-llms-2e5a93

Full version with abstracts: https://zhixudu.github.io/llms-full.txt

## Papers

- [What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute](https://zhixudu.github.io/projects/token-cost/index.md): Under review. arXiv:2610.02491.
  - Measuring per-token compute requirements of LLMs: If you work on adaptive or efficient inference, this paper defines a token's sufficient compute as the FLOPs of the cheapest panel model whose prediction reproduces that token given the correct prefix, giving an observed upper bound on what the token requires (Section 2, Definition 1).
  - Token-level mixture-of-agents as decision process: If you study model selection during decoding, this paper casts token-level mixture-of-agents as a Markov decision process under a compute budget and shows via a Bellman recursion that, along a fixed verified reference, the minimum-cost controller simply picks the cheapest matching agent at each position (Section 3, Eqs. 3-9, Algorithm 1).
  - Upper-bound semantics of reference reproduction cost: If you need formal guarantees for oracle compute measurements, this paper proves that summed per-token sufficient compute equals the minimum cost over selectors that reproduce a reference exactly and upper-bounds the cost of producing any correct response, and gives a local witness bound for larger model classes (Appendix B.2, Eq. 15).
  - Small models reproduce most reference tokens: If you work on small/large model collaboration, this paper finds that Qwen2.5-0.5B reproduces 91.9-95.2% of Qwen-72B reference tokens on GSM8K, MATH-500, and HumanEval, with OLMo-2 1B and R1-Distill-Qwen 1.5B at 89.34-91.74% and 89.80-92.27% (Section 4.1, Figures 1 and 3).
  - Heavy-tailed concentration of per-token compute: If you study where inference compute is spent, this paper shows the most expensive 10% of tokens account for 63.66-79.54% of estimated FLOPs in the Qwen panel, 63.75-69.37% in OLMo, and 67.19-74.76% in R1-Distill-Qwen, with the top 1% reaching 52.94-60.14% for Qwen (Section 4.1, Figure 3).
  - Cost conventions for unresolved tokens and Gini: If you worry that concentration results depend on how unmatched tokens are priced, this paper reports resolved-only top-10% shares of 51-60% (Qwen2.5), 40-44% (OLMo-2), and 26-32% (R1-Distill-Qwen) and resolved-only Gini coefficients from 0.218 to 0.547, with unresolved rates of 0.14-8.72% (Appendix C.1, Tables A5 and A6).
  - Shared token difficulty ordering across model families: If you study whether token difficulty is model-specific or shared, this paper finds that Qwen2.5, OLMo-2, and Falcon3 agree on which word positions need larger models, with partial Spearman correlations of 0.322-0.398 on Qwen-written and 0.417-0.475 on Llama-written references after controlling for surprisal and position (Section 4.1, Table 1).
  - Robustness checks on cross-family agreement: If you want to know whether shared difficulty survives confounds, this paper shows the association is positive on all twelve benchmarks and moves by less than 0.05 under extra controls, top-five and p>=0.5 criteria, a second verified response, memorization screening (33 of 1,758 items flagged), and GSM-Symbolic, and reaches 0.520 on Qwen3-30B-A3B MoE-generated text (Section 4.1, Appendix C.2, Tables A3 and A4).
  - Reproduction difficulty versus speculative acceptance signals: If you work on speculative decoding acceptance, this paper compares its reproduction indices with greedy and soft rejection proxies between each family's smallest and largest models and finds the reproduction ordering more shared across families (0.322/0.417 versus 0.150/0.199 greedy and 0.179/0.384 soft) (Section 4.1, Appendix C.2).
  - Surface features poorly predict expensive tokens: If you hope simple lexical cues identify hard tokens, this paper finds longer words weakly associated with higher reproduction demand and digits and operators weakly negative, with regressions on digit/operator indicators, word length, frequency, and position explaining only 2.5-3.2% of ranked variance (Section 6, Appendix C.2).
  - Teacher-forced agreement versus free-running continuation: If you study exposure bias or small-model handoffs, this paper shows that cheap reproduction under the correct prefix does not ensure reliable continuation: letting 0.5B generate before Qwen-72B resumes moves accuracy from -0.09 pp at one token to -9.44 pp at 64 tokens, and reference cost predicts handover loss only weakly (Spearman 0.195) (Section 4.2, Figure 4a, Table A8, Appendix D).
  - Mid-solution handover by capacity and start point: If you work on model relay or partial-solution handoff, this paper measures accuracy when 0.5B to 32B models complete a verified prefix from character fractions 0 to 0.9: 0.5B loses 54.08 points from the prompt and 37.64 from the midpoint, while 7B loses 1.50 points after 64 delegated tokens and 32B stays near zero (Section 4.2, Figure 4b, Table A9).
  - Oracle token routing versus confidence cascades: If you build token-level routers or cascades, this paper gives a routing controller oracle access to positions where 0.5B matches a 72B reference on all 500 MATH-500 problems, reaching 69.6% accuracy at 5.12 projected seconds versus 68.4% and 7.59 seconds for the best confidence-threshold cascade (theta = 0.99) (Section 4.3, Figure 5a, Tables A10 and A11).
  - Choosing speculative draft length from agreement maps: If you tune draft length in speculative decoding, this paper sets each draft block to the run of positions where 0.5B matches the 72B reference, proposing 301,691 versus 447,731 draft tokens (32.6% fewer) with similar accepted tokens and lowering projected latency from 4.98 to 3.99 seconds at 69.4% versus 69.0% accuracy (Section 4.3, Figure 5b, Appendix E.1).
  - Architecture-derived FLOP accounting for decoding: If you need a per-token FLOP model for dense and MoE transformers, this paper counts attention, projection, gated feed-forward, router, and vocabulary matrix multiplications with prefix-dependent attention cost, and its ledger shows drafting policies using more logical FLOPs (112.07 TFLOPs for map drafting, 154.30 for fixed) than 72B alone (101.92) despite lower projected latency (Appendix B.1, Table A10).
  - Latency projection from published throughput benchmarks: If you need latency estimates without a production deployment, this paper fits per-call and per-position costs to official Qwen2.5 vLLM A100 benchmarks (maximum relative error 2.13%) and tests 36 timing scenarios, under which map drafting stays faster than fixed-window and adaptive drafting by 8.3-24.7% and 7.3-18.4% (Appendix E.2).
  - Agreement with learned layer-skipping allocations: If you study adaptive-depth methods, this paper compares its reproduction indices with FlexiDepth's learned layer allocations on 600 generations, finding pooled partial correlations of 0.209 (Qwen panel) and 0.213 (OLMo panel), ranging from 0.061 on copy to 0.405 on product tasks (Appendix C.3, Table A7).
  - Verified-response corpus across twelve benchmarks: If you need a checker-verified token-level dataset across domains, this paper builds references from Qwen2.5-72B and Llama-3.3-70B on twelve benchmarks with 50,019 prompts spanning GSM8K, MATH-500, HumanEval, LiveCodeBench, MBPP+, AIME 2024, ARC-Challenge, BBH, MMLU, MMLU-Pro, HellaSwag, and NQ-Open, scored against Qwen2.5, OLMo-2, R1-Distill-Qwen, and Falcon3 panels (Section 4, Appendix A, Tables A1 and A2).
- [SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models](https://zhixudu.github.io/projects/sida-moe/index.md): MLSys 2024. arXiv:2310.18859.
  - Serving large MoE models under limited GPU memory: If you work on deploying Mixture-of-Experts models whose parameters exceed GPU memory, SiDA-MoE keeps only predicted-active experts on the GPU and holds the rest in host main memory, reporting up to 80% GPU memory saving on Switch Transformers (Section 4.1, Figure 8).
  - Predicting expert activation before inference: If you work on expert prefetching for MoE, this paper trains an offline predictor (called a hash function) that maps the input embedding sequence to the experts each token will use at every MoE layer, so experts can be loaded ahead of time instead of after each router call (Sections 3.2-3.3).
  - Replacing MoE routers with a learned predictor: If you study MoE gating, SiDA-MoE removes the routers from the forward pass entirely (they are offloaded to main memory) and substitutes the predictor's expert choices and scaling factors, while noting fixed routers such as Hash layers or BASE layers can be plugged in directly (Sections 3.1, 3.3).
  - Pipelining expert selection with inference: If you design MoE serving pipelines, the paper runs a hash-building thread that predicts expert tables for the next batch concurrently with the inference thread processing the current batch, with expert loading overlapped via pipeline parallelism and a FIFO expert cache under a memory budget (Section 3.1, Figure 5, Algorithm 1).
  - MoE inference overhead profiling: If you profile MoE inference cost, this paper measures that expert selection, expert invocation and communication take up to 72% of inference time on Switch-base-256 at batch size 1, growing with expert count, and notes that at small batch sizes the number of experts invoked dominates latency (Section 2.3, Figure 3, Remark 1).
  - Sentence-level expert activation sparsity: If you study expert usage patterns, the paper shows that even with load-balancing loss, a whole sentence activates fewer than 40% of experts in Switch-base-128 and fewer than 20% in Switch-base-256 on SST2, with idle ratios above 70% and 80% even for 80-token sentences (Section 2.4, Figure 4).
  - Effective GPU memory utilization of MoE: If you quantify memory waste in sparse models, the paper reports that MoE layers hold 78% to 99% of Switch Transformer parameters and that effective memory utilization can fall to about 5%, leaving roughly 24GB (Switch-base-128) and 50GB (Switch-base-256) idle on short SST2 inputs (Section 2.2, Table 2, Figure 2).
  - Sparse cross-token dependency of routing decisions: If you analyze what determines MoE routing, the paper corrupts or permutes random fractions of other tokens on C4 with Switch-base-128 and uses a combinatorial estimate to infer that a token's expert choice depends on only about 1 to 4 other tokens or positions (Section 3.4.1, Eq. 2, Figures 6-7).
  - LSTM with sparsemax attention predictor: If you need a lightweight sequence model that focuses on few inputs, the expert predictor is a 2-layer LSTM with self-attention whose weights use SparseMax, plus a residual connection from the current token, motivated by the sparse dependency finding (Section 3.4.2).
  - Truncated knowledge distillation of router logits: If you distill a large teacher's softmax into a low-capacity student, the paper computes the KD loss only over the teacher's top-T logits (T=30) and adds a cross-entropy term, because full-distribution KD degraded the small predictor (Section 3.5, Section 4 Hyperparameters).
  - Accuracy of expert activation prediction: If you need numbers on how predictable MoE routing is, top-3 hit rates of the predictor reach 99.00% (Switch-base-8, SST2) and 98.78% (Switch-base-128, SST2), dropping to about 90-92% on long MultiRC inputs (Section 4.6, Table 5).
  - Accuracy cost of approximate routing: If you assess the quality cost of predicted routing, fidelity is 97.5-99% for Switch-base-8 but 92.6-93% for Switch-base-128 on SST2/MRPC/MultiRC, and C4 perplexity rises notably (e.g. 6.68 to 18.49 for Switch-base-8, 4.59 to 8.11 for Switch-base-256), with larger models more tolerant (Section 4.4-4.5, Tables 3-4).
  - Throughput and latency versus DeepSpeed and Tutel: If you compare MoE inference systems, SiDA-MoE reports 2.60x and 3.93x throughput over the baseline average for Switch-base-128/256 on SST2, but only 1.01-1.04x for Switch-base-8, with gains shrinking on long MultiRC inputs (1.26x, 1.57x) and latency reduced to about 25% on SST2/MRPC and 60% on MultiRC (Section 4.2, Figures 9-10).
  - Expert offloading versus model parallelism under memory budgets: If you choose a strategy for a fixed GPU memory budget, the paper sweeps budgets from about 10 to 80GB on Switch-base-128/256 and finds data-aware expert offloading with a FIFO cache gives higher throughput than layer-wise model parallelism at every budget (Section 4.3, Figure 11).
  - Effect of input length on MoE memory savings: If you care how sequence length interacts with expert sparsity, memory reduction exceeds 80% on short SST2 sentences but falls to over 40% (Switch-base-128) and 20% (Switch-base-256) per the text for 200-500 token MultiRC inputs, since longer inputs touch more experts (Section 4.1, Figure 8).
  - Hierarchical GPU-CPU-SSD expert offloading: If you target trillion-scale MoE such as Switch-c-2048 (about 5TB), the discussion identifies host memory capacity as the limit and proposes GPU/main memory/SSD tiered expert offloading, plus a graph-structured predictor exploiting cross-layer activation dependencies (Section 6).
  - Positioning among MoE serving systems: If you survey efficient MoE inference, the related work contrasts SiDA-MoE's data-aware look-ahead prediction with communication-focused systems (DeepSpeed-MoE, Tutel, SE-MoE), task-enforced sparsity (M3ViT, Edge-MoE) and expert-caching schedulers (EdgeMoE, dynamic expert swapping), and states it is orthogonal to quantization and pruning (Section 5).
- [Rethinking Normalization Methods in Federated Learning](https://zhixudu.github.io/projects/fednorm/index.md): DistributedML @ CoNEXT 2022. arXiv:2210.03277.
  - External covariate shift in federated learning: If you study why FedAvg global models converge slowly, this paper names and defines external covariate shift: per-channel feature statistics diverge across clients because each runs independent local training, so neurons must re-adapt after every aggregation (Section 1, Section 3.1, Figure 1).
  - Why batch normalization fails in federated learning: If you work on BatchNorm in FL, this paper argues the accuracy drop is not only due to non-IID data: clients' channel statistics drift apart during local training, so averaged running statistics on the server mismatch the aggregated model's real feature statistics (Section 3.3).
  - Layer normalization as BatchNorm replacement in FL: If you are choosing a normalization layer for a federated CNN, this paper recommends LayerNorm because it normalizes per sample and has no cross-client running statistics; it gives the fastest convergence and best or near-best CIFAR-10 accuracy on CNN, VGG-11 and ResNet-18 (Section 3.4, Table 2, Figure 4).
  - Weight-norm imbalance obliterating client contributions: If you study aggregation fairness or client contribution in FedAvg, this paper shows with an FC-layer analysis that clients whose weights have much larger norms dominate the averaged model, erasing the contribution of small-norm clients (Section 3.1, Figure 2).
  - Scale invariance keeps client weight norms balanced: If you work on theory of normalization and auto learning-rate tuning, this paper uses the scale-invariance of BN/LN (gradient orthogonal to weights, Lemma 1) to show weight norms grow as initial norm plus accumulated squared gradients, so clients starting from the same initialization keep comparable norms (Section 2.2, Section 3.2, Eq. 5 and 7).
  - Two-device toy experiment on channel statistics: If you want direct evidence of client feature-statistics divergence, this paper trains two identical 3-layer single-channel conv models with BatchNorm on MNIST from the same initialization, with data differing only in mean, and shows the same channel ends up with clearly different output statistics on each device (Section 3.3, Figure 3).
  - Comparing BN, GN, LN, IN and no normalization in FedAvg: If you need a benchmark of normalization layers under FedAvg, this paper compares group, layer, instance, batch, fixed (non-aggregated) batch normalization and no normalization on CIFAR-10 with VGG-11, ResNet-18 and a LeNet-style CNN over 5000 rounds and three seeds (Section 4, Table 2).
  - Group normalization performs close to layer normalization: If you use GroupNorm in FL following Hsieh et al., this paper finds GroupNorm tracks LayerNorm closely in all three architectures (e.g., 63.10% vs 63.52% on VGG-11), and notes GroupNorm as instance-based normalization may be sensitive to sample noise (Section 4.2, Table 2, Section 5).
  - Instance norm and non-aggregated BatchNorm statistics underperform: If you consider keeping BatchNorm statistics local (not aggregated) or using InstanceNorm, this paper finds fixed BatchNorm is no better than standard BatchNorm (e.g., 33.85% vs 34.38% on ResNet-18) and InstanceNorm trails LayerNorm on all three models (Table 2).
  - ResNet-18 without normalization in federated learning: If you train residual networks in FL, this paper reports that ResNet-18 with no normalization reaches 60.73% versus 59.70% for LayerNorm and 34.38% for BatchNorm, attributing this to residual connections keeping weights near initialization, but notes it needs careful hyperparameter tuning for stable training (Section 4.2, Table 2).
  - Pathological non-IID CIFAR-10 FedAvg setup: If you replicate label-skew FL experiments, this paper uses CIFAR-10 split across 100 devices with 2 classes and 100 samples per class each, 10 clients sampled per round, 10 local epochs, batch size 64, SGD lr 0.01, 5000 rounds, on TITAN RTX GPUs (Section 4, Section 4.1).
  - Normalization effects on federated convergence speed: If you care about communication rounds to convergence in FL, this paper's accuracy-vs-rounds curves show LayerNorm converging fastest across CNN, VGG-11 and ResNet-18, with BatchNorm lagging and VGG-11 LayerNorm gaining 11.96 and 27.01 points over BatchNorm and no normalization (Section 4.2, Figure 4).
  - Rethinking FedBN, HeteroFL static BN and GroupNorm in FL: If you survey normalization alternatives for FL, this paper positions FedBN (local BN, limited to personalized FL), GroupNorm (Hsieh et al.) and HeteroFL's static BatchNorm, arguing none explains why BatchNorm fails and offering external covariate shift as that explanation (Section 5).
- [MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing](https://zhixudu.github.io/projects/moe-gps/index.md): NeurIPS 2025 Workshop (ML for Systems). arXiv:2506.07366.
  - Choosing expert load predictors for MoE inference: If you design dynamic expert duplication for multi-GPU MoE inference, this paper gives MoE-GPS, a framework that picks the prediction strategy (distribution-only vs token-to-expert, and at what accuracy) that minimizes simulated end-to-end latency for a given model and hardware setup, summarized as a two-question decision flow (Figure 1, Section 4).
  - Distribution-only prediction of expert load: If you predict MoE expert popularity, this paper proposes predicting only the aggregate token share per expert (a per-layer multinomial fit by MLE, computed offline with no runtime overhead) and shows it gives a 23% speedup over the best token-to-expert predictor at skewness 1.4 on Mixtral 8x7B (Section 3.2.1, Section 4, Appendix A).
  - Accuracy vs overhead of token-to-expert routing predictors: If you train predictors of which expert each token will be routed to, this paper shows the accuracy-overhead tradeoff gives a U-shaped end-to-end latency curve, so the best predictor sits at intermediate accuracy rather than the most accurate one (Figure 4, Figure 6b/6d).
  - Interconnect bandwidth decides the prediction strategy: If you serve MoE models over PCIe or NVLink, this paper finds token-to-expert prediction pays off mainly when communication dominates (e.g., PCIe 4.0), because it lets tokens skip the scatter step, while distribution-only prediction is preferred when communication is not the bottleneck (Figure 6, Figure 7, Section 4 takeaways).
  - Routing skewness metric and its effect on prediction: If you measure MoE load imbalance, this paper defines skewness as tokens on the most popular expert divided by the average per expert, shows FFN compute and communication time scale with it, and finds higher skewness makes token-level prediction cheaper but distribution estimates less accurate (Section 2, Table 1, Section 4).
  - Expert routing skew across MMLU, AlpacaEval, SST2: If you want empirical expert activation statistics, this paper measures Mixtral 8x7B average skewness of 1.388 (MMLU), 1.402 (AlpacaEval) and 1.990 (SST2) at sequence length 512, with distribution-estimation error rates of 1.80%, 0.98% and 16.00% (Section 3.2.1, Table 1).
  - Greedy expert duplication and placement algorithm: If you implement expert replication for load balancing, this paper gives an iterative algorithm that copies the most-used expert from the most-loaded GPU to the least-loaded one under memory and maximum-copy limits and reassigns tokens until per-GPU loads differ by at most one (Algorithm 1, Section 3.1).
  - Modeling prediction error impact on runtime: If you need to translate predictor error into system slowdown, this paper models optimistic, typical (bottleneck GPU at (1+e) times average) and pessimistic (N(1+e)) error placement for FFN compute, and notes misrouting always adds communication (Figure 5, Section 3.3).
  - Simulating MoE expert parallelism with LLMCompass: If you evaluate MoE serving designs without a real cluster, this paper extends the LLMCompass architectural simulator with expert parallelism communication, Mixtral support (GQA, SwiGLU, sliding-window attention) and tunable predictor accuracy/overhead, and simulates single-layer prefill latency on four A100s (Section 3.4).
  - Lightweight neural predictors of expert routing: If you build expert routing predictors, this paper compares a global-frequency baseline, token- or position-conditioned frequency models, a small 2-layer MLP and an LSTM with sparse attention trained on token embeddings with per-layer heads, and notes MLP predictors hit an accuracy floor at long sequences while LSTMs parallelize poorly (Section 3.2.2, Appendix B, Section 5).
  - Load imbalance in MoE prefill under TP plus EP: If you study MoE inference parallelism, this paper analyzes prefill with tensor parallelism for attention and expert parallelism for FFN, showing per-GPU communication grows as (N-1)·skewness/N^2 and FFN time scales with skewness, and argues decode load balancing is secondary (Section 2, Figure 2).
  - Hiding expert weight transfer behind attention: If you worry about the cost of moving duplicated experts, this paper estimates one Mixtral 8x7B FP16 expert transfer takes about 0.1 ms over NVLink 3.0, smaller than attention runtime at batch 1 and sequence 512, and that over PCIe 4.0 it can be hidden with modest workloads such as batch 16 and sequence 2K (Section 5).
  - Generalization to Llama-MoE and Switch Transformer: If you use MoE models other than Mixtral, this paper repeats the simulations on Llama-MoE and Switch Transformer with the same datasets and hardware, finding similar trends, higher skewness, and prediction cost that grows exponentially as accuracy approaches perfect (Appendix C, Figures 8 and 9, Section 5).
  - Prefill latency breakdown of Mixtral 8x7B: If you profile MoE layers, this paper breaks simulated single-layer Mixtral 8x7B prefill latency into attention, attention communication, FFN, FFN communication and predictor overhead across skewness levels, showing communication dominates under PCIe (Figure 6a/6c).
  - Positioning among expert duplication systems: If you survey MoE load balancing work, this paper situates prediction-driven duplication relative to MoE-Prediction, Prophet, FlexMoE, SE-MoE and FasterMoE, training-time auxiliary loss and expert biasing, and expert offloading approaches, treating the latter as orthogonal (Section 2, Section 3.1).
- [AUDITA: certified auditing and causal attribution of adverse outcomes in autonomous multi-agent systems](https://zhixudu.github.io/projects/audita/index.md): arXiv preprint 2026. arXiv:2608.22160.
  - Graded failure attribution for LLM multi-agent systems: If you work on automated failure attribution for LLM multi-agent pipelines, this paper replaces single-culprit labels with a per-agent graded responsibility verdict and, on live GSM8K planner-solver-aggregator-verifier pipelines, cuts the Who&When-style LLM judge's responsibility error from 0.589 to 0.170 (gold-informed) and 0.159 (no answer key) (Results, Table 1).
  - Most multi-agent failures are jointly caused: If you study why multi-agent LLM systems fail, this paper finds that under duty-aware graded ground truth 95-98% of adverse incidents (56/57, 43/47, 39/41) are joint solver-aggregator incidents, none fit the single-culprit structure, and in 58.7% of Who&When traces a duty-bearing verifier or orchestrator stayed silent through the decisive error (Results, live failures).
  - Halpern-Pearl actual causality for agent messages: If you apply actual causality or degree-of-responsibility theory to AI systems, this paper builds a structural causal model over certified inter-agent messages and duty-indexed absence variables, grades causes with the modified Halpern-Pearl definition and the Chockler-Halpern degree rho = 1/(|X*|+|W*|), and handles overdetermination (1/2, 1/2 split), preemption and omission natively (Methods, Eqs. 1-2).
  - Production gate before counterfactual grading: If you need to separate causal production from counterfactual dependence, this paper adds a gate that retires commands whose committed effect never propagated to the outcome before any counterfactual is computed (Proposition 2), and shows the gate is load-bearing on preemption and on a transcript-identical 'inert twin' that every transcript-level method misclassifies (Ablations; Extended Data Fig. S1).
  - Tamper-evident signed logs for agent accountability: If you design audit logs for AI agents, this paper specifies a message format (author, payload, citation set, committed effect predicate, Ed25519 signature) sealed into Merkle-committed append-only epochs, proves forgeries leave the verdict unchanged and deletions only widen a responsibility interval (Proposition 3), and measures that forgeries drive an unsigned log to 0.365 error and frame an innocent in all 450 probes while the certified verdict stays at 0.000 (Figure 3b).
  - Framing attacks by insiders with valid keys: If you study adversarial manipulation of blame or responsibility, this paper constructs two conduct-level attacks (conditioning and inevitability rigging) that make a compliant agent a true but-for cause, proves such an agent is never found culpable (Theorem 1) and that a replay-checkable blame-shift certificate grades the framers (Theorem 2, extended to all orders in Theorem 5); over 600 incidents victim involvement reaches 1.0 but culpability stays 0.000, and a level-2 scapegoat convicts a benign deviator in 0 of 330 draws (Results, Robustness).
  - Limits of ground-truth-free auditing: If you ask what an auditor can certify without an answer key, this paper proves a completeness barrier: when compliant agents can make honest errors, no breach standard computed from the record is both sound and complete, and closing the gap amounts to verifying task correctness (Theorem 4); empirically the gold-free deployable verdict matches the gold-informed one (0.159 vs 0.170) (Ablations; Remark S8).
  - Negligence doctrine and NESS test for AI blame: If you connect legal causation to AI accountability, this paper outputs a per-agent duty-breach-causation-harm verdict from negligence law that can report a negligent-but-inert agent or an involved-but-not-culpable conduit, and proves that on monotone incident models its causes coincide with the NESS test from tort doctrine via minimal transversals of prime implicants (Remark 1; Proposition S1; Figure 2).
  - LLM judges blame someone even with no culprit: If you evaluate LLM-as-a-judge for incident analysis, this paper shows that on 300 no-cause accidents LLM judges named a culprit in 100% of runs across all three Who&When formats, even with a prompt offering NONE, and an anomaly detector did so in every case, while the certified engine manufactured a culprit in zero cases (Results, Negative controls).
  - CulpaBench benchmark from robot accident records: If you need a benchmark with analytic ground truth for responsibility attribution, this paper releases CulpaBench, over 1,000 incidents in a deterministic multi-robot facility whose structures (redundancy, inert twin, preemption, omission, delegation chain, record attack, no-cause and trivial controls) are grounded in 42 public OSHA/NIOSH robot-accident records, with ground truth from an independent oracle sharing no code with the engine (Table 2; Extended Data Table 1; Note S5).
  - Responsibility for omissions and unexpected startup: If you work on blaming inaction, this paper encodes required-but-unissued commands as first-class absence variables and, on 74 held-out unexpected-startup incidents (the modal fatal pattern in US robot fatality data) where the only breach is an omitted halt, is exact while single-site counterfactual scoring blames the innocent sole actor every time and LLM judges convict an innocent in 67-100% of runs (Results, Generalisation).
  - Shapley value versus causal responsibility for agents: If you use Shapley values for multi-agent credit or blame, this paper finds Shapley the strongest baseline on live pipelines (0.223) but duty-blind on CulpaBench (average 0.452), and the comparison depends on model scale: Shapley wins on Qwen2.5-7B (0.041 vs 0.111) while the graded verdict leads on Llama-3.1-8B and Qwen2.5-72B, with margins -0.070, +0.037, +0.053 (Figure 3a; Table 2).
  - Bitwise-exact LLM replay for counterfactual certification: If you need reproducible counterfactual replay of LLM agents, this paper certifies each reported cause by re-execution under a serial replay protocol on pinned open-weight models in vLLM (concurrent batching breaks bitwise exactness even in batch-invariant modes) and, for hosted non-deterministic models, gives a Hoeffding-based guarantee bounding false causes by alpha plus an action-match score (Proposition 1; Note S7).
  - Verdicts depend on the declared duty standard: If you study normative standards in AI accountability, this paper shows that holding causal facts fixed and varying only the registered duty standard flips culpability in 69.8% of cases, and that removing the duty layer raises live error from 0.170 to 0.408, so a verdict must ship with its standard attached (Results, Ablations).
  - Attribution on static logs: Who&When comparison: If you benchmark on Who&When, this paper reports a degraded structure-aware attributor without replay or cryptography reaching agent-accuracy 0.408 and step-accuracy 0.321 on the 184 static traces, within the Qwen2.5-72B judge band (0.41-0.51) and above two of three judge formats on step accuracy, arguing the advantage requires counterfactual access (Table 3).
  - Computational cost of exact actual causation: If you worry about the NP-hardness of actual causation in practice, this paper measures exact recovery at every tested incident width while wall-clock per query roughly doubles per added candidate (0.02 s at k=2 to 62.1 s at k=14), placing the practical limit near k=8 at a one-second budget, with record overhead of 393 B per signed message and 118 KB per incident (Extended Data Table 2; Costs and limits).
  - Off-record influence, flooding and re-chunking robustness: If you ask how attribution behaves outside its assumptions or under manipulation, this paper moves 31% of coordination off the recorded bus and finds 0.157 of responsibility becomes invisible with over-attribution staying 0.000 over 1,200 draws, shows flooding the log with inert commands has slope 0.000 for all methods, proves grades are invariant to how an agent splits its output into messages (Proposition S3), and finds verdict-guided repair averts 2.4x the harm of a random fix (Costs and limits; Note S6).
  - Accident investigation and regulation for embodied AI fleets: If you work on incident investigation, liability or AI regulation for LLM-commanded robot fleets, this paper proposes what EU AI Act-mandated event records must be (signed, cited, sealed, causally analysable) to support accountability, positions itself relative to STAMP/CAST blame-free analysis, the ethical black box, PeerReview and Accountable Virtual Machines, and lists its scope limits: discrete threshold incidents only and simulated, not physical-robot, evidence (Discussion; Note S10).
- [Swimba: Switch Mamba Model Scales State Space Models](https://zhixudu.github.io/projects/swimba/index.md): NeurIPS 2026. arXiv:2603.06938.
  - Mixture-of-experts inside SSM token mixers: If you want to add MoE to the sequence-mixing layer of a state space model rather than to the FFN, this paper introduces Swimba, which replaces the Mamba-2 in-projection with a set of expert projections plus a token-level router (Section 4.1, Fig. 3).
  - Taxonomy of MoE-SSM designs: If you are comparing ways to combine MoE with state space models, this paper separates MoE over separated SSMs (one state trajectory per expert, compute grows with experts) from MoE-parameterized SSMs (experts mixed in parameter space, one trajectory) and argues prior work conflates them (Sections 1, 3.2-3.3, Fig. 2).
  - Single recurrence cost independent of expert count: If you worry that MoE in a recurrent layer replicates the scan, this paper proves the parameter-mixed layer costs O(T C_step + T C_mix) versus O(T E C_step) for separated experts, so the recurrence is evaluated once regardless of the number of experts (Theorem 2, Appendix A.2).
  - Parameter-space mixing preserves SSD structure: If you need MoE routing to stay compatible with Mamba-2's SSD kernel, this paper shows that mixing expert injection and readout streams yields exactly one selective SSM with state size N, independent of the number of experts and of top-k sparsity, so the existing SSD implementation is reused unchanged (Theorem 1, Section 3.1).
  - Stability of expert switching in recurrences: If you are concerned that token-wise expert switching destabilizes recurrent dynamics, this paper gives a BIBO stability bound: with a contractive shared transition (norm of A at most rho < 1) and bounded mixed injection and readout, state and output stay bounded (Theorem 3, Appendix A.3).
  - When mixed and separated expert SSMs coincide: If you study whether averaging expert parameters approximates running separate expert recurrences, this paper proves exact equality when routing weights are time-invariant and readout is shared, and bounds the output gap otherwise by a stable recursion driven by each expert's deviation from the mixture injection (Theorem 4, Appendix A.4).
  - Expressivity gain from routed linear projections: If you ask whether parameter-space MoE adds modeling power or is only an efficiency trick, this paper proves that with softmax routing over linear expert projections the layer strictly contains single-expert layers, via a two-expert construction that realizes the logistic sigmoid, which no single-expert layer (a degree-3 polynomial in this setting) can represent (Theorem 5, Appendix A.5).
  - Shared transition, expert-specific B, C and X: If you are choosing which SSM parameters to make expert-specific, Swimba shares the transition A across experts and time and places expert diversity only in the injection and readout streams (B, C, X), which keeps one coherent state trajectory (Section 3.2, Eq. 7-9; Section 6).
  - Upcycling a pretrained hybrid Mamba-Transformer: If you want to add experts to an existing pretrained hybrid model cheaply, this paper initializes from the Nemotron-H-8B checkpoint, replaces every Mamba-2 layer with a Swimba layer in the 52-layer hybrid layout, freezes the copied backbone, and trains only new expert and router parameters with SFT on the Llama-Nemotron Post-Training Dataset (Sections 4.2-4.3).
  - Matched-FLOPs accuracy on LM benchmarks: If you need evidence that SSM-layer MoE helps at constant compute, Swimba-14B (4 experts, top-1 per layer) raises the average accuracy from 69.47 to 70.81 (normalized 70.98 to 72.34) over Nemotron-H-8B across ARC-Challenge, ARC-Easy, BoolQ, HellaSwag, MMLU, OpenBookQA, PIQA, RTE and WinoGrande, e.g. MMLU 71.7 to 75.0, while BoolQ and OpenBookQA dip slightly (Table 1).
  - Per-token FLOPs of top-1 MoE SSM layers: If you measure the compute cost of sparse experts in SSM layers, this paper uses the PyTorch profiler and finds Swimba-14B at 1.51282e10 FLOPs per token versus 1.51263e10 for Nemotron-H-8B, a difference under 0.2% (Table 2, Section 5.2).
  - vLLM serving cost of MoE-SSM models: If you deploy SSM or hybrid models in vLLM, this paper benchmarks decoding throughput and latency on a single 40GB A100 across batch sizes 1-64 and input-output lengths 256-2048, finding Swimba-14B slightly slower than Nemotron-H-8B, with as low as a 10% slowdown (Fig. 4, Fig. 5, Section 5.2).
  - FLOPs versus wall-clock gap from routing: If you study why MoE models run slower than their FLOPs suggest, this paper reports that despite near-identical FLOPs Swimba shows a larger real-time gap, which it attributes to routing overhead, and notes that adding experts with a fixed number of active experts should leave throughput and latency largely unchanged (Section 5.2).
  - Positioning against MoE-Mamba, BlackMamba, Jamba, Routing Mamba: If you survey MoE-SSM hybrids, this paper contrasts block-mixing approaches that interleave dense SSM mixers with MoE MLPs (BlackMamba, MoE-Mamba, Jamba, Zamba, Nemotron-H/Nemotron-3) and Routing Mamba's MoE projections, with its own parameter-space mixing that routes the B_t and C_t governing the state update (Section 6).
  - MoE for token mixing beyond attention: If you work on mixture-of-experts attention such as SwitchHead or Mixture-of-Attention-Heads, this paper adapts the idea of routing within token-mixing layers to linear-time SSMs, motivated by keeping O(L) mixing cost while growing parameters (Fig. 1, Section 6).
  - Link to switching state-space models: If you know classical switching state-space models (Ghahramani and Hinton), this paper relates them to MoE over separated SSMs, where each expert keeps its own memory, and contrasts this with compressing expert dynamics into one shared trajectory (Section 1, Section 3.4).