Research

Model Test Results: Round One

November 18, 2026. Results and need-to-know facts only.

What was tested

Nine models, the workhorse tier from nine companies: Anthropic Sonnet 5, OpenAI GPT-6.1 Sol, xAI Grok 4.3, GLM 5.2, Google Gemini 3.8 Flash, Mistral Medium 3.5, DeepSeek V4 Flash, Kimi K3, Qwen 3.7 Plus.

30 tasks, all document work: 8 decode, 7 extract, 5 draft, 4 compare, 4 answer questions, 2 summarize. Every document was synthetic, made for the test.

All nine models were run through OpenRouter, a single API that carries models from every company in this test. One access point kept the conditions identical for every model, and every call returned its exact cost and token usage, which is what the cost figures in this report are built on. Settings were the same for every model, including temperature zero.

Same instructions and same conditions for every model, including the same answer budget of 1,200 tokens. Answers were graded blind against a fixed answer key. The grader did not know which model wrote which answer.

Grader: Claude Opus 5. The grader is an Anthropic model and an Anthropic model finished first, so the grading was hand-checked: 60 of the 269 grades, including every Anthropic answer, were re-read against the answer key. No grade found would change any model's standing.

Each model attempted each task once. Scores a few points apart are within noise. Large gaps are not.

Results overall

ModelScoreCost, 30 tasksAnswer time (middle)
Anthropic Sonnet 599.5%$0.14667.2s
OpenAI GPT-6.1 Sol96.1%$0.06445.4s
xAI Grok 4.395.5%$0.05148.8s
GLM 5.292.7%$0.05544.8s
Google Gemini 3.8 Flash90.6%$0.10388.4s
Mistral Medium 3.590.3%$0.03871.8s
DeepSeek V4 Flash89.9%$0.00353.2s
Kimi K373.3%$0.38467.7s
Qwen 3.7 Plus33.0%$0.045520.8s

Results by category

Category sizes: Decode 8 tasks, Extract 7, Draft 5, Compare 4, Questions 4, Summarize 2. Five of the six categories tied at the top, so the category winners are shared. At this size the top group cannot be separated by category. The only category with a single winner was Questions.

CategoryBestModels at the best scoreLowest
Decode100%GPT-6.1 Sol, Sonnet 5Qwen 3.7 Plus, 0%
Extract100%Grok 4.3, Kimi K3, Sonnet 5DeepSeek V4 Flash, 86.0%
Draft100%GLM 5.2, Kimi K3, Mistral Medium 3.5Qwen 3.7 Plus, 15.2%
Compare100%Sonnet 5, Gemini 3.8 FlashQwen 3.7 Plus, 23.8%
Questions94.4%Sonnet 5Qwen 3.7 Plus, 13.9%
Summarize100%GPT-6.1 Sol, Grok 4.3, Mistral Medium 3.5Qwen 3.7 Plus, 50.0%

Qwen 3.7 Plus finished last in five of the six categories.

Findings

28 of 270 answers came back blank. The model used its whole answer budget on internal reasoning and wrote nothing. Qwen: 18 of 30. Kimi: 8 of 30. DeepSeek: 2 of 30. Blank answers scored zero.

One answer picked a trap date planted in the test documents (DeepSeek, extraction). It scored zero on that task.

Mistral lost one task to a rate-limit error. It scored zero and was not retried.

One DeepSeek answer received no readable grade after three attempts and was left out of its score. Counted as zero, DeepSeek would be about 87% instead of 89.9%.

Cost of this round: $0.89 for the models' answers, about $2.62 for grading, about $3.52 total.

The 1,200-token answer budget skewed the bottom of the table. Reasoning models spend from the same budget to think, and two of them ran out before writing. Scored only on answers actually written: Kimi K3 95.8% (73.3% overall), Qwen 3.7 Plus 73.7% (33.0% overall), DeepSeek V4 Flash 93.0% (89.9% overall). The top six models returned no blank answers and were unaffected.

A fixed token budget is not a fair budget. DeepSeek cost about 42 times less per task than Sonnet, and Mistral about 4 times less while answering 4 times faster. Either could run a much larger token budget and still cost less money and time than a rival inside the cap.

Kimi K3 is the opposite case, and a bust on cost in both directions. It was the most expensive model in the round at $0.3846 for 30 tasks, about 2.6 times Sonnet's cost per task and about 110 times DeepSeek's, and it still left 8 of 30 answers blank. Under an equal-cost budget it would receive fewer tokens than the others, not more.

What was not tested

Meta's three models (Wonder, Spark, Ember). Meta offers them only through its own API, not through the marketplace used here, and the test bench cannot reach that API yet. They will be tested through Meta's API under the same blind grading.

Flagship and light-tier models. Not in this round. The models in this round were not the latest releases from their companies.

Anything other than document tasks.

The next test

Models: the latest flagship, workhorse, and light models from ten companies. 33 models in total.

Tasks: 161. 87 agent tasks and 74 document tasks.

Total graded runs: 5,313.

Grading is by category, not just overall, on three measures: speed, quality, and cost. Cost is counted two ways: dollars per task to us, and app tokens per task to the customer. The app's tokens are priced from the dollar cost, not from raw token counts.

Every model is tested under two budgets: a fixed token budget, which shows what breaks under a hard cap, and an equal-cost budget, the same dollars per task for every model, which measures quality per dollar.

When a newer model is known to beat an older one, the newer one replaces it and the older one is not tested again. Already applied: GPT-6.2 Astra replaced GPT-6 Astra, GPT-6.2 Sol replaced GPT-6.1 Sol, Qwen4-Max replaced Qwen 3.8 Max.

Open Dejargon