Touchstone, a system for AI-assisted network automation, checks answers from small local language models before returning them. A September 25 arXiv paper by Maleeha Masood and Momina Nofal reports that it kept most requests local on some structured network tasks while sending uncertain cases to GPT-5.5.
Seven off-the-shelf models, each with 1 billion to 8 billion parameters, propose answers. Touchstone rejects candidates that fail a task-specific test, such as a network policy that both permits and denies the same route. It weighs the remaining candidates and returns a high-confidence result locally; when the evidence is weak, it sends the request to GPT-5.5. Passing a check is not proof of correctness: a test catches only errors it was designed to detect.
On 500 conflict-detection examples, Touchstone reached 98.6% accuracy and escalated 16% of requests. The comparison sent every input to GPT-5.5, which scored 100%.
Intent translation asks the system to turn plain-language network requirements into JSON rules for reachability, waypoints and load balancing. In the paper's example, “Traffic originating from Barcelona can reach subnet 100.0.9.0/24” becomes a reachability rule for that location and prefix.
Across 500 such examples, Touchstone reached 93.8% accuracy and escalated 17% of requests, versus 99.2% accuracy when every input went to GPT-5.5. The check's false-accept rate was 5.6%: among intent-translation responses it admitted locally, that share was still incorrect.
The authors also tested 500 TeleQnA questions, which lack a task-specific test for a wrong answer. Touchstone scored 78.2% while escalating 16%; GPT-5.5 scored 84.2%. A well-formed answer can still be wrong when the task offers no simple way to check it.
The authors evaluated benchmark datasets, not live production networks. Escalated requests still go to GPT-5.5, so the system does not keep every input local. Its results suggest that small local models are most useful when task-specific checks can catch errors, but neither these scores nor the checks establish deployment safety.
Sources
- Original paper: Can You Check That? The Checkability Boundary for Local LLM Network Automation
- Full paper and evaluation tables
- Server-rack image source and public-domain record
Image credit: Federal Bureau of Investigation, public domain, via Wikimedia Commons. The image is illustrative.