Researchers at the University of Alberta and the Alberta Machine Intelligence Institute have released LongHarness, a benchmark for testing how software around AI models affects long-context work. The top of 25 tested model-and-harness setups, GPT-5.6-sol with mini-swe-agent, scored 68% exact-match accuracy when the four task-suite scores were averaged equally; no setup led all four.
Illustration: Figure 2 from Pham, Nguyen, Chen and Ye’s LongHarness paper (2026), reused under CC BY 4.0.
A harness is the software that controls a model’s context, tools and sequence of calls. Instead of sending a whole document for one read, it can search for terms, retrieve related passages, run a script or ask the model to work through smaller steps.
The four suites have 50 instances each. They ask models to find people who meet several conditions scattered across documents; match functionally equivalent Python programs; trace an eight-step query through prior computations; and identify memo claims that conflict only when combined with facts elsewhere. The project describes the contexts as near 128,000 tokens, with sizes varying by task.
The researchers compared five language models both directly and with four harnesses: RLM, OpenCode, mini-swe-agent and ReAct. Each evaluated instance had a cumulative ceiling of three million input and output tokens across calls. Answers were scored by exact match, while dollar costs were estimates based on the authors’ stated API rates.
The best tool depends on the task. For GPT-5.6-sol on program tracing, OpenCode scored 96% exact match versus 94% with direct inference, while estimated cost rose from $0.648 to $1.37 per instance. In Outlier Memo Detection, each of 50 instances contains 700 memos, exactly 15 of which make conflicting claims. GPT-5.6-sol with mini-swe-agent returned the correct set on all 50 instances, for 100% accuracy at an estimated $0.583 per instance; the same model with direct inference scored 44% at $1.17.
These are the authors’ initial results in a new arXiv preprint. The tasks are constructed English-language cases and Python programs, not natural user work; strict exact-match scoring counts partial answers as misses, public answer keys create a contamination risk, and dollar figures depend on the stated price assumptions. Independent replication is still needed.
LongHarness gives teams a way to compare a model together with the retrieval and tool layer that directs it. Follow-up tests on naturally occurring tasks will show whether the task-specific accuracy and cost trade-offs persist outside the benchmark.
Sources
- LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning, arXiv preprint posted 29 September 2026; methods, task definitions, results and limitations.
- LongHarness project page, with the authors’ results dashboard and task descriptions.
- Official code and scoring repository and full benchmark dataset.