| Date | 23 Sep 2026 |
|---|
An artificial intelligence coding agent can write a patch that sails through ordinary tests and still fall apart the moment a real model is loaded and a server starts taking requests. That gap is the point of SWE-Serve, a benchmark Nvidia researchers described Tuesday on the company’s developer blog.
The suite covers 53 tasks built from 83 merged pull requests in SGLang, an open-source system used to serve large language models. Nineteen of those tasks start a live server and run the patch through the full serving path. On that slice, the same 627 patches passed 45.9 percent of the time with the complete verifier. Drop the live-serving checks and the rate jumped to 69.4 percent. One hundred forty-seven patches flipped from fail to pass. Roughly one in three that cleared everything else still failed once the server was up.
Jennifer Williams, Dave Farris, Jeff Farris and Jiantao Jiao wrote the post. Williams is a senior researcher on Nvidia’s Nemotron data, scaling and evaluation team. Dave Farris is a principal research engineer on the same team. The work was developed with input from the SGLang group, listed as a launch partner.
SWE-Serve is not another general coding contest. Existing repository benchmarks lean toward ordinary software work. Inference tests often stop at kernel generation or speed tweaks. This one asks agents to make repository-scale changes across the serving stack: model enablement, decoding, caching, scheduling, serving APIs and runtime performance.
The 53 tasks split into six families. Speculative and advanced decoding accounts for 14. Model and backend enablement accounts for 12. Kernels, quantization and performance, plus serving APIs and runtime correctness, have eight apiece. Caching and runtime state has seven. Distributed execution and scheduling has four. Twelve tasks run on CPU. Forty-one use a single Nvidia H100. This first release does not cover other inference engines, multi-GPU runs or multi-node serving.
Thirty-seven tasks come from one upstream pull request. The other 16 stitch together two to six related changes. Each task hands the agent an instruction and a containerized SGLang checkout from before the target change. The patch passes only if it satisfies a hidden verifier on the declared hardware. It is never scored against the reference implementation.
These are not one-line edits. The median reference solution touches 553 lines across seven files. A typical verifier packs seven tests for the new behavior and 10 regression tests. Three tasks also enforce a calibrated performance gate on an H100.
One example asks the agent to add serving support for dense and mixture-of-experts Qwen3.5 models, starting from a revision that has none. Both the 0.8B dense model and the 35B-A3B MoE model have to load and serve through ordinary SGLang interfaces on one H100. The verifier checks registration, configuration and weight loading, image and video inputs, OpenAI-compatible requests, native batched generation, log probabilities and execution through the MoE model’s routed experts.
Live serving is where a lot of work dies. Those 19 tasks contain 276 live-serving tests. Two hundred forty-two were sourced or adapted from SGLang. The other 34 cover behavior introduced by the merged changes when no fitting upstream test existed. On the Gemma 4 MoE task, 16 of 33 patches passed every other check and still failed at least one live-serving test. Those tests cover model loading, expert routing, text and image serving, and batched generation with correct ordering and log probabilities.
A SWE-Serve pass is narrow. It means the patch satisfied the benchmark verifier. It is not SGLang’s review process. It does not mean the patch is deployable, ready to merge or blessed by maintainers.
Agents also slip when the work crosses more of the runtime. Researchers split the request-to-output path into four domains: request handling and I/O; scheduling and request lifecycle; model execution; and KV-cache and runtime-resource management. Across each of the 11 models’ best settings, the 26 single-domain tasks posted a 69.0 percent pass rate. The 27 multi-domain tasks posted 47.7 percent. That is 21.3 percentage points. Every model setting moved in the same direction.
Scores themselves are all over the map. Eleven models and 31 model-effort configurations were run with mini-swe-agent, a minimal agent that uses only Bash, under closed-book conditions. Mean pass@1, using each model’s best tested setting, ran from 34.6 percent to 75.5 percent. Claude Opus 5 and GPT-5.6 Sol both reached 75 percent at maximum reasoning effort across three full runs of the 53 tasks. Claude Opus 5 averaged $17.40 and 57.5 minutes per task. GPT-5.6 Sol averaged $12.26 and 29.5 minutes. Claude Sonnet 5, Kimi K3, GPT-5.6 Luna and GPT-5.6 Terra tied at 64 percent, with mean cost from 95 cents to $7.24 and wall time from 25.5 minutes to 99.9 minutes. Inkling S landed at 35 percent.
Each session was capped at 210 minutes and 350 steps. Native harnesses did not lift the two leaders. GPT-5.6 Sol scored 73.6 percent in Codex. Claude Opus 5 scored 69.8 percent in Claude Code. Both posted 75.5 percent with mini-swe-agent. No model led all six engineering families.
The team screened 786 potential sources, built 156 executable candidates and kept 53. On declared hardware, the unmodified repository had to fail the new-behavior tests and still pass regressions. A reference patch had to clear the full verifier. Evaluations blocked the public web and upstream source repositories. Hugging Face stayed open for model weights after an open-network pilot showed models pulling task-specific upstream code. Researchers audited all 1,749 leaderboard trials. One hundred ninety-six prohibited retrieval attempts were blocked. None succeeded.
SWE-Serve’s environments, verifiers and baseline configurations are posted with a leaderboard and a paper on arXiv.