Debate between copies of the same language model can change a group answer in either direction: it may correct a wrong majority or overturn a right one. In 6,925 closed-book multiple-choice debates on MMLU-Pro, a new study counted 253 cases of the latter, showing why an evaluation that reports only final accuracy can miss both reversals and useful corrections.
Each test used three copies of one model. They first answered independently, then saw one another’s answers and debated for three rounds. The final group answer was the majority. The researchers called a change from a correct starting majority to a wrong final one a “collapse,” and the reverse a “correction.”
The team also replayed a rule that stopped debate for model setups flagged by a short pre-debate screen of how readily they changed answers. In a separate held-out set of 6,525 debates, the rule prevented 29 of 251 collapses but also blocked 108 of 804 corrections. With the two outcomes weighted equally, that trade-off amounted to 1.21 percentage points lower accuracy in that subset.
The result applies to closed-book multiple-choice questions debated by copies of the same model; it does not establish how often mixed-model panels or tool-using agents would fail. The authors released code and an open data tier so other researchers can score debate rules on both the reversals they prevent and the corrections they lose.
Sources
- Li, Liu and Yuen: Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
- Authors’ DebateLedger code and data repository
- Cover figure: Li, Liu and Yuen, from the paper, CC BY 4.0.