Lab notes · 04 · November 2026
When Fable reviews, Qwen fixes more
On 80 real issues from open-source projects, a published 4-bit fine-tune of Qwen3.8 27B, served on my own hardware, resolved 65 of the 78 that ran cleanly (83%). With Claude Fable 5.2 reviewing its first patches, for about $0.40 a task at API list prices, it resolved 74 (95%), fixing 9 and breaking none. In a follow-up, telling Qwen only that a patch was incomplete fixed 2 of the 11 real failures; Fable’s full review fixed 9. The gain needs a second run, and each run had twice the benchmark’s time limit, and Fable may have seen these public fixes in training.
By Robert Balas, Redcom Software
- Tasks
- 80 of the 642 SWE-bench Pro V2 tasks, drawn at random within each repository before any run; hidden tests decide whether a patch resolves the issue
- Worker
- A published 4-bit fine-tune of Qwen3.8 27B (not trained by me) on a local DGX Spark, in a coding-agent harness, 99 minutes per run (the benchmark allows 49)
- Helper
- Claude Fable 5.2 in short read-only sessions: reads the issue, the repository and, for a review, Qwen’s patch; never sees the hidden tests; has no shell; may describe changes but not write them
- Result
- Qwen alone 65 of 78 (83%); with Fable’s review 74 of 78 (95%; 9 fixed, 0 broken, p = 0.004); with Fable’s brief 71 of 75 (95%; 89% if its 5 missing tasks count as failures); Qwen reviewing itself 63 of 78 (81%)
- Cost
- $0.40 per task for a review and $0.45 for a brief, at API list prices; the local hardware is not priced
- Follow-up
- 41 of the tasks rerun with other help: the Fable brief repeated (8 fixed, 0 broken); Opus 5.5 and Sonnet 5.5 briefs did about as well at 50% and 18% of the cost; a verdict without Fable’s notes fixed 2 of 11 real failures, the full review 9
- Status
- Version 2: one run of the review arm, two of the brief arm (the second on 41 tasks); the run data is not public yet
The question
Open models that run on one machine can now resolve real issues in real repositories, slowly and not always. Frontier models score higher on public leaderboards, but every task sent to them costs money. A middle path is to let the local model do the work and ask the frontier model only for advice: a short brief before it starts, or a review of its first attempt. In this design the frontier model still reads the repository, so the middle path may lower cost, but it does not keep the code private.
I wanted to know whether that advice makes the local model solve more, and whether the gain comes from what the frontier model writes or simply from a second try. So I ran the same 80 tasks five ways, with two controls. The local model ran twice on its own, to measure how much a rerun changes by chance. It also reviewed its own patch, to separate a second look from a better reviewer.
A follow-up on 41 of the tasks then asked which part of the help matters and whether it has to come from Fable. It tried shorter briefs, briefs written by three other Claude models, and a review verdict with the reasons left out.
The setup
Tasks. 80 of the 642 tasks in SWE-bench Pro V2, drawn at random within each repository, with the seed fixed before any run. Each task is a real issue from one of 11 open-source projects, and a hidden test suite decides whether a patch resolves it.
Worker. A third-party fine-tune of Qwen3.8 27B, quantised to 4 bits. It runs on a DGX Spark I own (Nvidia’s desktop AI computer), served by sglang, an open-source model server. It works through dsh, a coding-agent harness that can drive any model, inside each task’s container. Harbor, a benchmark runner, starts the containers and grades the patches. The containers and the harness run on a separate client computer that calls the Spark over the network; two such computers were used in turn.
Helper. Claude Fable 5.2 in short, read-only sessions. It sees the issue text and the repository, and for a review also Qwen’s patch. It has no shell and no network, and it never sees the hidden tests or the reference fix. It is told to describe changes, not write them: no diffs, and no code block longer than three lines.
Arms. The same 80 tasks were run five ways, called arms (Figure 1). Each has a full name and a short one.
| Arm | Short name | What Qwen gets |
|---|---|---|
| Qwen alone, pass 1 | pass 1 | The task only. |
| Qwen alone, pass 2 | pass 2 | The task only, run again with identical settings, to measure run-to-run noise. |
| Qwen reviews itself | self-review | Its own pass-1 patch on every task, with an instruction to check it against the issue and fix it. No Fable. |
| Qwen + Fable brief | Fable brief | The task plus Fable’s brief, about 500 words: where the problem is, its cause, the approach, the pitfalls and how to check the fix. |
| Qwen + Fable review | Fable review | Fable first reviews the pass-1 patch and answers APPROVE or REVISE. On REVISE, Qwen gets that patch again with the review attached. On APPROVE, nothing changes and the patch keeps its pass-1 result. |
Time. Each run had 99 minutes of Qwen’s time, about twice the benchmark’s own 49, because the Spark slows down when it serves several runs at once. Container setup, Fable’s sessions and the final test run are not counted.
Terms. A task is resolved when its hidden tests pass. It is graded in an arm when at least one attempt there ran cleanly to the end; a few have no result in some arms (see Discarded attempts). Between two arms, a task is fixed when it goes from failing to passing and broken when it goes the other way.
- Qwen works on the task
- Hidden tests
- Pass-1 patch
- Qwen checks and fixes it
- Hidden tests
- Fable writes a brief
- Qwen works from it
- Hidden tests
- Pass-1 patch
- Fable: approve or reviseAPPROVE keeps pass 1
- If REVISE: Qwen again
- Hidden tests
What happened
Resolve rate
| Arm | Graded | Resolved | Rate | 95% interval | Within 49 min |
|---|---|---|---|---|---|
| Qwen alone, pass 1 | 78 | 65 | 83.3% | 73.5–90.0% | 50 (64.1%) |
| Qwen alone, pass 2 | 76 | 64 | 84.2% | 74.4–90.7% | 52 (68.4%) |
| Qwen reviews itself | 78 | 63 | 80.8% | 70.7–88.0% | 40 (51.3%) |
| Qwen + Fable brief | 75 | 71 | 94.7% | 87.1–97.9% | 60 (80.0%) |
| Qwen + Fable review | 78 | 74 | 94.9% | 87.5–98.0% | 48 (61.5%) |
The intervals are Wilson intervals. The last column applies the benchmark’s 49-minute limit to all of Qwen’s time on a task. For self-review and the review arm, that time includes the pass-1 run they start from. On that basis the brief arm still leads. The review arm falls just below pass 1 (61.5% against 64.1%): a review buys accuracy with a second run, and the second run takes time (Figure 2).
The same tasks, with and without help
Rates on slightly different sets of tasks can mislead, so each comparison below uses only the tasks graded in both arms. The p-value comes from an exact paired test (McNemar’s), which asks whether the tasks that changed went mostly one way. All four comparisons with Fable fall below the usual 0.05 threshold. Running six tests makes a chance result more likely, so with a Bonferroni correction (threshold 0.008) only the two review comparisons stay below it (Figure 3).
| Comparison | Tasks | Resolved | Fixed | Broken | p |
|---|---|---|---|---|---|
| Pass 1 → pass 2 (noise) | 75 | 63 → 63 | 4 | 4 | 1.000 |
| Pass 1 → self-review | 78 | 65 → 63 | 1 | 3 | 0.625 |
| Pass 1 → Fable brief | 74 | 62 → 70 | 9 | 1 | 0.021 |
| Pass 2 → Fable brief | 73 | 62 → 69 | 8 | 1 | 0.039 |
| Pass 1 → Fable review | 78 | 65 → 74 | 9 | 0 | 0.004 |
| Self-review → Fable review | 78 | 63 → 74 | 11 | 0 | < 0.001 |
Two identical solo passes disagreed on 8 of 75 tasks, 4 each way, so a shift of a few tasks is noise. Against that, the review arm fixed 9 and broke none. On the 72 tasks graded in every arm, the resolved counts are 61 in each solo pass, 59 for self-review, 68 with a brief and 69 with a review.
The brief’s evidence is weaker than its rate. Five tasks have no brief-arm result, because every attempt was discarded (see Discarded attempts). Pass 1 had resolved three of them. Counted as failures, they bring the brief arm to 71 of 80 (88.8%), and its comparison with pass 1 to 9 fixed and 4 broken (p = 0.267), which is no longer distinguishable from noise. One of the five was force-killed at the time limit on every attempt, which is close to a failure anyway. The review arm has no such gap: it has a result for every task that pass 1 has. A second brief run in the follow-up shows the same pattern (see Follow-up).
- without Fable
- with Fable’s help
- no-result tasks counted as failures
- within 49 minutes
- without Fable
- with Fable’s help
- noise seen between pass 1 and pass 2
Was Fable’s review right?
Mostly, with many false alarms (Figure 4). Fable caught 11 of the 13 pass-1 patches that failed, and 46 of the 48 it approved did pass. But 19 of its 30 REVISE verdicts were on patches that already passed. Rerunning all 30 with the review attached fixed 9 and broke none, so here the false alarms cost time and nothing else.
What Fable wrote mattered more than the second try. On the 11 patches it marked REVISE that really failed, a plain second attempt resolved 3 of 10 graded, Qwen reviewing itself 1 of 11, a run with Fable’s brief 8 of 10, and a run with Fable’s review 9 of 11 (Figure 5). Self-review also did harm where there was little to fix: on the 48 patches Fable approved, 46 of which already passed, it broke 3 and fixed none. The review arm leaves approved patches alone, so it cannot break them. The follow-up tested whether the verdict alone would do as well; it did not (Figure 9).
- failed the tests
- passed the tests
2 failed · 46 passed
11 failed · 19 passed
Follow-up: which part of the help matters
After the main runs, I ran 41 of the 80 tasks again with other kinds of help. The 41 are the 17 tasks Qwen failed alone in pass 1 or pass 2, 20 drawn at random (seed 20261115) from the 57 it solved both times that also had a brief-arm result, and the 5 with no brief-arm result, one of which is also among the 17. The set is harder than the full sample by design, so its rates cannot be compared with the 80-task rates above. The worker, the harness, the 99-minute limit and the rule for discarded attempts are the same as in the main runs.
Six kinds of brief were tested (Figure 6). Three come from Fable: its original briefs, given to a fresh Qwen run; a short note from a new Fable session limited to five tool calls and 150 words; and the original brief cut down to its first two parts, where and cause. The other three are full briefs from Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, written with the same instructions as Fable’s. A seventh arm tested the review. Qwen restarted from each of the 30 patches Fable had marked REVISE and was told only that a senior engineer had judged it incomplete, without Fable’s notes.
Each comparison below uses the tasks graded in both arms. The p-values are not corrected for multiple comparisons, and with 41 tasks only large differences can be told from noise.
- written by Fable
- written by another Claude model
- part left out
The brief result repeated
The same Fable briefs, given to a fresh Qwen run, fixed 8 tasks and broke none against pass 1 (p = 0.008). The first brief run had fixed 9 and broken 1 on these tasks (Figure 7). The two brief runs agree on 33 of the 35 tasks graded in both. The repeat again has tasks with no result, 4 of them. Counted as failures, they turn its comparison with pass 1 into 8 fixed and 2 broken (p = 0.109). So in neither brief run does the gain survive counting the missing tasks as failures.
Less guidance helped less
The full brief has five parts: where, cause, approach, pitfalls and how to check. Both shorter versions keep only where and cause. Against pass 1, the short note fixed 8 and broke 2, and the cut-down brief fixed 7 and broke 4. Both fixed about as many tasks as the full brief and broke more that Qwen had solved alone. Against the full brief on the same tasks, the gaps are a few tasks and within noise (Figure 8).
- no brief
- Fable’s brief
- another model’s brief
- noise between the solo passes
Briefs from other Claude models
Against pass 1, Opus 5.5 briefs fixed 7 and broke 1, Sonnet 5.5 briefs fixed 8 and broke none, and Haiku 5.5 briefs fixed 5 and broke 2 (Figure 7). On the tasks graded in both, the Fable brief resolved 32 to Opus’s 31, 32 to Sonnet’s 32 and 33 to Haiku’s 28 (Figure 8).
Opus and Sonnet briefs did about as well as Fable’s. On these 41 tasks a Fable brief cost $0.48 on average, an Opus brief $0.24 and a Sonnet brief $0.09, about 50% and 18% of Fable’s price. Haiku briefs cost $0.01 and helped less. Qwen resolved 28 tasks with them where it resolved 33 with Fable’s, the largest gap in the follow-up, though on 41 tasks it is still within noise (p = 0.125).
- Fable’s brief
- another model’s brief
A verdict without the notes did no better than self-review
On the 30 tasks whose pass-1 patch Fable marked REVISE, Qwen was told only that its patch had been judged incomplete. It resolved 20 of the 30, one more than pass 1 and the same as when it reviewed itself. The bare verdict fixed 2 and broke 1. With Fable’s notes, the same tasks went from 20 to 28 resolved (p = 0.008). Of the 11 patches that had really failed, the bare verdict led to 2 fixes and the full review to 9 (Figure 9). The content of the review is what helps.
- passes
- fails
- broken: passed in pass 1
- fixed without Fable’s notes
- fixed with them
· 1 broken
Cost
| Arm | Fable sessions | Fable cost | Per task | Net tasks gained | Cost per task gained |
|---|---|---|---|---|---|
| Qwen reviews itself | 0 | $0.00 | $0.00 | −2 | – |
| Qwen + Fable brief | 80 | $36.08 | $0.45 | +8 | $4.51 |
| Qwen + Fable review | 78 | $31.30 | $0.40 | +9 | $3.48 |
Fable’s cost is the API list price of each session, as reported by the session itself. Net tasks gained is fixed minus broken against pass 1, on the tasks graded in both. All 80 briefs are charged, including those for tasks with no brief-arm result. Counting those five tasks as failures, the brief’s net gain is +5, or $7.22 per task gained. Another $10.31 went on briefs and reviews thrown away after setup mistakes and is not in the table. Qwen ran on hardware I own, so its time is not priced: about 43 hours of graded runs for pass 1, 47 for pass 2, 37 for the brief arm, 22 for self-review and 21 for the 30 review reruns, the last two on top of pass 1. Discarded attempts are not counted.
Brief-arm runs averaged 29.5 minutes against 33.0 and 36.9 for the two solo passes. That is not evidence that a brief makes Qwen faster. The two identical solo passes differ by 3.9 minutes, about as much as the brief arm differs from pass 1, and the brief average leaves out its five tasks with no result.
The follow-up’s new briefs cost $10.14 for the 41 short notes, $9.80 for Opus 5.5, $3.57 for Sonnet 5.5 and $0.39 for Haiku 5.5. The repeat run reused the original Fable briefs, and the cut-down briefs needed no new session. Follow-up run times are not compared here: several arms shared the Spark at once, and the load changed from hour to hour.
Discarded attempts
An attempt is discarded when the harness itself fails: the connection to the model server drops, the harness crashes or has to be force-killed, or the starting patch cannot be applied. The rule is the same in every arm and does not look at the test result. Discarded attempts are retried, and a task with no clean attempt has no result. A run that reaches the time limit and exits cleanly counts as a result.
| Reason | Pass 1 | Pass 2 | Self-review | Brief | Review |
|---|---|---|---|---|---|
| Connection to the model server dropped | 20 | 13 | 0 | 5 | 0 |
| Harness crashed | 3 | 2 | 0 | 2 | 0 |
| Harness force-killed at the time limit | 2 | 0 | 0 | 2 | 0 |
| Harness exited with an error | 0 | 1 | 0 | 1 | 0 |
| Starting patch did not apply | 0 | 0 | 3 | 0 | 6 |
| Tasks left with no result | 2 | 4 | 2 | 5 | 2 |
Self-review and the review arm have no result for the two tasks with no pass-1 patch to start from. All their discarded attempts were retried successfully.
The dropped connections were partly explained during the follow-up. The model server logged no error when they happened, and its proxy recorded them as closed by the client. The main client computer reached the model server over a VPN, and the VPN client rebinds its connections whenever Docker removes a finished task’s network. That timing matched one burst of four drops, not every drop, and it does not explain the drops seen with the other client computer. Separately, 23 runs made while one client computer was overloaded were set aside whatever their result, and those tasks were rerun on the other.
The follow-up arms, under the same rule:
| Reason | Fable repeat | Short note | Where and cause | Opus 5.5 | Sonnet 5.5 | Haiku 5.5 | Verdict only |
|---|---|---|---|---|---|---|---|
| Connection to the model server dropped | 4 | 4 | 14 | 8 | 12 | 8 | 0 |
| Harness crashed | 2 | 2 | 4 | 2 | 2 | 4 | 0 |
| Harness force-killed at the time limit | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| Harness exited with an error | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| Task setup failed or timed out | 1 | 5 | 7 | 0 | 0 | 16 | 15 |
| Tasks left with no result | 4 | 2 | 2 | 4 | 3 | 1 | 0 |
Most of the setup failures came from too many tasks starting at once when an arm was launched. Some timed out during setup; I stopped others by hand within minutes to bring the load down (12 at the first launch, 10 at the second). All were retried.
What I take from it
- Here, a short review from a stronger model did far more than a second attempt. On the 11 failing patches Fable sent back, its review led to 9 fixes; a plain retry managed 3 of 10 and a self-review 1 of 11.
- Fable’s reviews raised many false alarms, 19 of 30 here. That is cheap only when, as in this design, a false alarm leads to a rerun rather than to a rejected change.
- What helped was the content of the advice. Told only that its patch was incomplete, Qwen fixed 2 of the 11 real failures; given Fable’s review notes, it fixed 9.
- The advisor need not be the most expensive model. On the follow-up tasks, Sonnet 5.5 briefs did about as well as Fable’s, at about a fifth of the price, on one run of 41 tasks. Haiku 5.5 briefs did worse, by a margin 41 tasks cannot separate from noise.
- Cutting a brief to where and cause kept most of the fixes but broke more of the tasks Qwen could already do.
- A gain measured with a generous time limit can vanish under a strict one. Under the benchmark’s 49 minutes the review arm gained nothing, and only the brief arm still came out ahead, by a margin not tested for significance.
What this does not show
- Fable may know these fixes. The tasks come from public repositories, and Fable may have seen the real patches in training. On private code the gain could be smaller. This is the largest open question.
- Qwen may know them too. The worker is a third-party fine-tune, and I do not know its training data. That would lift every arm, so it bears on the absolute rates more than on the comparison between arms.
- The briefs are prescriptive. They name the files, functions and behaviour needed, so the result is closer to Fable diagnosing and Qwen implementing than to Qwen reasoning better.
- The sample is small. 80 tasks, with one run of the review arm and two of the brief arm, the second on 41 tasks. The intervals are wide, and in neither brief run does the gain survive counting the missing tasks as failures. The follow-up comparisons between kinds of brief rest on 41 tasks and one run each.
- The brief repeat reused Fable’s briefs. It shows that Qwen’s gain with the same briefs holds on a second run, not that a second set of Fable briefs would do as well.
- The rates depend on the doubled time limit, and the review on a second run. Each run had 99 minutes, about twice the benchmark’s limit, and a reviewed task that was sent back got a second run on top of the first. At the benchmark’s 49 minutes the rates are 64.1% for Qwen alone in pass 1 (68.4% in pass 2), 80.0% with a brief and 61.5% with a review.
- One worker, one harness, one benchmark. Published scores from other setups are not comparable with these numbers.
- Fable was never run on these tasks alone, by choice. So this shows the pairing beats Qwen alone; it does not show how Fable alone would score, or how much cheaper the pairing is than using Fable for everything.
Next
This second version adds the follow-up, which asked which part of the help matters and whether it needs Fable. The two questions I most want answered are still open: whether the gain holds on fixes Fable cannot have seen in training, and how the pairing compares with Fable working alone.
Related work
The pattern is not new. Anthropic’s advisor tool, Aider’s architect mode and a memo from Fireworks in June 2026 all pair a cheaper worker with a stronger advisor. The Fireworks memo advised open-weight workers with Claude Opus 4.8 on three benchmarks, including 60 SWE-bench Pro tasks, where it reported gains of 4 to 7 points. Across its six benchmark-and-worker experiments, a review alone did as well as or better than a plan (their version of a brief) plus a review in five, and a reviewer of the same model gave no gain. This run also finds no gain from Qwen reviewing itself, nor from a bare verdict without the review’s notes, and finds the brief and the review about equal: on the 74 tasks graded in both, the review resolved 2 that the brief did not, and the brief 1 that the review did not (p = 1.000). Its gain over Qwen alone is larger, 11 to 12 points, but only under the doubled time limit. The worker, the advisor, the tasks and the time limit all differ, so the figures are not directly comparable.
Where the evidence is
Every run’s logs, patch, token use and test output are kept, as are Fable’s briefs and reviews with the cost of each session, the exact instructions Fable was given, the task lists, the follow-up briefs with the cost of each session, a log of what was started when during the follow-up, and the script that computes every number in this note from the graded runs. They are not public yet.