Redcom Software · lab
Lab notes
Research I run alongside client work, on local models, agents and evaluation. Each note gives the method and the numbers, including the results that went against me. Every number comes from recorded data; run summaries are available on request.
- When Fable reviews, Qwen fixes moreOn 80 real issues from open-source projects, a published 4-bit fine-tune of Qwen3.8 27B, served on my own hardware, resolved 65 of the 78 that ran cleanly. When Claude Fable 5.2 reviewed each first patch and sent back the ones it doubted, Qwen resolved 74, fixing 9 and breaking none. A follow-up found that Sonnet 5.5 briefs did about as well as Fable’s at 18% of the cost. The gain needs a second run, and each run had twice the benchmark’s time limit, and Fable may have seen these public fixes in training.
Result · 78 tasks, one square each
- 65 resolved by Qwen alone
- 9 fixed after Fable’s review
- 4 still unresolved
none broken · p = 0.004
- A local model counted the corrections. A hand count disagreed.A 27B model on the author’s own DGX Spark, behind masking and a sandbox, read 20,174 prompts from fourteen months of Claude Code history on one of the author’s machines overnight and counted one request in eight as a correction. A blind hand count by the author on a small 2026 sample found nearly one in four, and the model missed most of those; Claude Opus 5.5 did little better. The hand check stopped a number that may be too low from being reported unchecked. The hand count hints at a fall late in 2026 and the model’s rate edges down, though the samples are too small to be sure.
Result · correction rate
Local model, whole history12.5%Hand count, 44 requests from 202623%95% interval 13–37% · on the same 44 the model flagged 6, 2 in common
- When in doubt, the agent doesn’t payBefore an AI agent pays for an API call over x402 (pay-per-request inside HTTP), a gate I built checks the live price and payee against the service’s listing and the buyer’s rules, and never takes missing evidence as a yes. In fixture tests on 40 synthetic providers, 35 built to misbehave, every one whose live terms drifted from its listing was denied and all five clean controls were allowed; nothing is live yet.
One search · 23 listings matched
- 18 denied
- 2 inconclusive
- 3 allowed
as recorded at the release review; not checked one by one
- Sharper on one dataset, reversed on the nextA side network 0.2% the size of a frozen language model read four of its layers and made its answers more decisive than a same-size control did, at all three fresh seeds tested. On a second dataset the effect reversed and no change to generated text could be shown, so I stopped under a rule written before the deciding runs.
Result · change in margin
Development setreader aheadSecond corpuscontrol ahead- distinct reader
- same-size control
4 cells each · second corpus: gate held open