A few weeks ago, I published the results from building Patchwork—a local, self-healing code repair agent that uses LangGraph, Ollama, and an isolated Pytest sandbox to repair Python bugs on a 4GB laptop GPU.

In that first version, a 4-bit qwen2.5-coder:3b model hit a 58% overall fix rate across a 100-defect benchmark, jumping from a 38% single-shot baseline thanks to its test-driven reflection loop. But it also uncovered a glaring issue: a 17-point confirmation bias gap, where the model wrote weak self-tests that passed on its own broken code 67% of the time.

After staring at the failure logs, I realized something uncomfortable: several failures weren’t the model’s fault at all.

My own agent scaffolding had subtle bugs, blind spots, and sandbox leaks that were tripping the model up. So I set out to fix the plumbing in Patchwork v1.1 and answer two practical engineering questions:

  1. How much can cleaner scaffolding actually improve a 3B model?
  2. What happens if we keep the exact same agent harness and scale to a 7B model on the same 4GB laptop GPU?

Here is what broke, how I fixed it, and what the data showed.

1. The Scaffolding Bugs Hiding in Plain Sight

When an autonomous agent fails, it is tempting to blame the model weights. But when I profiled the v1.0 pipeline, I found four real bugs in my own harness:

Blind Spot 1: The Reflection Prompt Amnesia

In v1.0, when tests failed, the reflection node sent the model the error traceback and its previous patch:

"Failure diagnostics: AssertionError: assert 5 == 6"
"Current Code: def add(a, b): return a - b"

Notice what was missing:

  1. The actual test code that failed. The model was told line 8 failed with 5 == 6, but had to guess what assertion was actually written on line 8.
  2. The original baseline code. Without the original code as an anchor, each retry caused “code drift”—the model would slowly delete docstrings, strip helper functions, or mutate unrelated logic.

The Fix: I enriched _build_reflect_prompt to provide structured context:

  • The exact failing test assertions.
  • The candidate patch under test.
  • The original buggy code as an immutable baseline reference.

Blind Spot 2: Python’s Hidden Underscore Import Trap

In the test sandbox, we executed generated tests against the patch using:

from module_under_test import *

In Python, wildcard imports silently ignore any identifier that starts with an underscore (_meets_min_length, _qualifies_for_discount).

Several boundary-condition defects were failing with:

NameError: name '_meets_min_length' is not defined

The model’s repair was completely correct, but the test sandbox literally couldn’t see the private validation helpers!

The Fix: Replaced the wildcard import with dynamic symbol mounting:

import module_under_test

for _sym in dir(module_under_test):
    globals()[_sym] = getattr(module_under_test, _sym)

Now private module helpers, inner constants, and utility functions are 100% accessible to generated test suites.

Blind Spot 3: Subprocess Explosion in Unit Testing (7m08s → 50s)

Running pytest tests/ took over 7 minutes on Windows. Profiling revealed that TestEvaluateAll in test_evaluate.py executed all 100 benchmark defects against live subprocesses (Ruff linter, self-test runner, oracle runner), spawning over 300 processes during every single test run.

The Fix: Mocked the per-defect evaluation function in the unit test suite, isolating the test harness aggregation logic from the subprocess execution layer. Total test suite duration dropped from 428 seconds down to 50 seconds (an 88% reduction) with zero loss in test coverage.

Blind Spot 4: Worker Thread Starvation in LLM Invocations

_llm_timeout_executor was initialized with ThreadPoolExecutor(max_workers=1) and a 45-second future.result(timeout=45.0). When an inference call hung, future.result() timed out in the caller, but the background thread remained blocked waiting for Ollama/httpx.

The next queued defect immediately timed out because the single worker was still occupied.

The Fix: Pushed timeout enforcement down to the HTTP transport layer:

ChatOllama(model=model, client_kwargs={"timeout": 45.0})

And caught httpx.TimeoutException at the socket boundary, aborting the connection and freeing the executor worker immediately.

2. Why Did 3B Only Go from 58% to 60%?

After fixing all those scaffolding bugs, sanitizing unicode escapes, and handling Ollama connection refusals gracefully, I re-ran the entire 100-defect benchmark on qwen2.5-coder:3b.

The result: 60% overall pass rate (up from 58%).

You might ask: Why did fixing all those real bugs only rescue two extra defects?

This was one of the most important takeaways of the project:

  • What scaffolding can fix: Better reflection prompts and sandbox fixes give the model the right context to fix syntax-level mistakes, remember its original code, and retry without getting stuck. That is why boundary condition and zip truncation defects flipped to PASS.
  • What scaffolding cannot fix: Qwen 2.5 Coder 3B is a 3-billion parameter, 4-bit quantized model running locally on a laptop GPU. For categories like float precision (IEEE 754 rounding), regex character escapes, and generator state exhaustion, the model simply does not understand the underlying runtime mechanics.

When an SLM fundamentally does not know how floating-point math or generator iterators work in Python, feeding it the traceback just makes it generate different broken code.

Cleaner agent scaffolding makes the repair loop reliable and prevents crashes, but it cannot magically give a 3B model reasoning capabilities that are not in its weights.

3. The Scaling Experiment: Testing with Qwen 2.5 Coder 7B

To prove whether the remaining 40% of failures were agent harness flaws or 3B model limits, I added a --model flag and ran the exact same 100-defect benchmark through qwen2.5-coder:7b (benchmarks/results_7b.json).

Running a 7B Model on a 4GB Laptop GPU

An RTX 3050 Laptop GPU has ~4GB of physical VRAM. A 4-bit 7B model is ~4.7 GB.

Ollama handled this through partial layer offloading: loading ~65% of the model layers into the GPU’s VRAM and offloading the remaining layers to system RAM (CPU).

  • Peak VRAM: Capped cleanly at 2377 MB (~2.4 GB). Zero out-of-memory crashes.
  • Average Latency: 24.09s per defect (vs. 8.45s on 3B). The entire 100-defect sweep finished in about 40 minutes.

The Benchmark Comparison

Metric Initial Run (v1.0 - 3B) Improved Run (v1.1 - 3B) Scale Test (7B)
Pass@1 (First Try) 38% 38% 60% (+22)
Pass@Overall (within 3 retries) 58% 60% 76% (+16)
Confirmation Bias Gap 17 points 9 points 5 points
Average Time / Defect 7.80s 8.45s 24.09s
Peak GPU VRAM 2532 MB 2513 MB 2377 MB (~2.4 GB)

The jump was immediate:

  1. Pass@1 skyrocketed from 38% to 60%: The 7B model got 60 out of 100 defects right on its first try without needing a single retry—matching 3B’s final score after 3 full reflection retries.
  2. Overall pass rate reached 76%: It repaired 76 out of 100 defects.

4. What 7B Cracked Open vs. What Still Failed

Comparing the per-category breakdown between 3B and 7B revealed exactly where parameter scaling matters:

Category                      | 3B Pass | 7B Pass | Delta
---------------------------------------------------------
regex_special_chars           | 0 / 5   | 3 / 5   | +3
string_strip_misuse           | 0 / 5   | 3 / 5   | +3
shallow_deep_copy             | 1 / 4   | 4 / 4   | +3
off_by_one_slicing            | 3 / 5   | 5 / 5   | +2
incorrect_boundary_conditions | 2 / 5   | 4 / 5   | +2
generator_exhaustion          | 0 / 5   | 0 / 5   |  0 (Both failed 100%)
float_precision               | 0 / 5   | 1 / 5   | +1

The Breakthroughs:

  • regex_special_chars (0% → 60%): 7B correctly understood raw regex string escapes (r"\[.*\]" and re.escape).
  • string_strip_misuse (0% → 60%): 7B recognized that .strip(".txt") strips character sets, and correctly replaced it with Python 3.9’s .removesuffix(".txt") and .removeprefix().
  • shallow_deep_copy (25% → 100%): 7B properly imported copy and applied copy.deepcopy() to nested mutable structures.
  • off_by_one_slicing (60% → 100%): Fixed all slicing bounds cleanly.

The Stubborn Blind Spot (0% on Both Models):

  • generator_exhaustion (0/5 on 3B and 0/5 on 7B): Both models completely failed every single generator defect.

    When code iterates over a generator expression twice (sum(gen) followed by len(list(gen))), both models treat the variable like a reusable list. Neither model conceptually understands Python generator exhaustion lifecycle without explicit prompt steering or static AST linting.

This proves that certain runtime semantics cannot be solved simply by jumping from 3B to 7B parameters.

5. The Confirmation Bias Collapse (17 Points → 5 Points)

In the 3B run, self-written unit tests created a huge false-positive gap: self-tests passed on 75% of defects, but the real oracle only passed on 58% (a 17-point gap where the model wrote weak tests that validated its own broken code).

With 7B, that gap shrank to just 5 percentage points (81% self-test pass vs. 76% oracle pass).

Why? The 7B model writes much stricter, higher-fidelity assertions. Instead of writing trivial happy-path tests that rubber-stamp broken code, it wrote strict boundary checks that actually failed when the patch was defective, giving the reflection loop a real opportunity to self-heal.

What’s Next: Patchwork-Verify

All of this brings us to the core problem of autonomous code repair:

In the real world, you don’t have a hidden oracle test suite to grade your agent. How do you know if an SLM’s patch is actually correct without an oracle?

If self-tests have a false-positive bias, how can we build cheap, oracle-free verification signals to catch false accepts before deploying a patch?

That is the question behind my next project, Patchwork-Verify: evaluating lightweight verification signals like:

  • Signal V1 (Discriminating Tests): Does the generated test pass on the patch AND fail on the original buggy code? (If it passes on both, the test is blind to the bug).
  • Signal V2 (AST Mutation Score): Do generated tests kill syntactic mutants of the patch?
  • Signal V3 (Dual Execution Consensus): CodeT-style output agreement across sampled candidates.

The code and benchmark results for Patchwork v1.1 are open source on GitHub (tagged at v1.1.0), and I’m excited to share the verification findings next.