MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference

要約

Temperature-zero BF16 LLM inference is often treated as reproducible, yet the same request can emit different tokens when decoded alone or inside a larger batch. Existing fixes use batch-invariant operators or LLM-42’s per-token verification, incurring cost even when most steps are stable. We ask wh…