Can an edge-quantized 3-billion-parameter language model running on an entry-level laptop GPU autonomously repair real software bugs?

When I started building Patchwork, the primary objective was deterministic local execution: an offline, self-healing code auditing agent that could run entirely on a 4GB laptop GPU (an RTX 3050 Laptop GPU) under 2.5GB of VRAM, without sending proprietary source files to commercial cloud APIs.

To evaluate whether an agentic reflection loop actually improves a small code model—or merely burns compute repeating the same errors—I constructed a 100-defect benchmark across 21 fault domains, pairing each defect with an unseen oracle test.

Here is what the empirical data revealed about small language model (SLM) reasoning, state-graph orchestration, and the hidden danger of self-generated test confirmation bias.

1. System Architecture: LangGraph State Machine

Patchwork runs on top of LangGraph with local inference served through Ollama using qwen2.5-coder:3b (4-bit quantized).

Rather than treating repair as a single unconstrained text-generation prompt, the graph isolates execution into discrete deterministic stages:

[Buggy Python File]
        │
        ▼
[Static Analysis: ast.parse + ruff]
        │
        ▼
[Patch & Test Generator (Qwen 2.5 Coder 3B)] ──► Pydantic Schema Validation
        │
        ▼
[Sandboxed Test Runner (subprocess, timeout, tempdir)]
        │
   ┌────┴────────────────────────┐
   ▼                             ▼
Tests Fail                  Tests Pass
(retries < 3)                    │
   │                             │
   ▼                             ▼
[Reflect Node] ────────► [Compile Audit Report]

Eliminating Silent Hallucinations with Pydantic

Small models frequently leak markdown commentary, emit invalid syntax, or accidentally concatenate generated unit test declarations directly into production function bodies.

To enforce deterministic boundaries, LLM responses are strictly parsed into a Pydantic schema:

from pydantic import BaseModel, Field

class CodeAuditOutput(BaseModel):
    bug_analysis: str = Field(description="Terse diagnosis of the defect")
    patched_code: str = Field(description="Pure Python code containing the fix")
    test_suite: str = Field(description="Pytest unit tests verifying the fix")

If the generated code contains markdown backticks, invalid indentation, or fails ast.parse(), it is rejected immediately at the boundary and fed back into the graph’s error handler before any subprocess is spawned.

2. Benchmark Results: Measuring the Reflection Dividend

Each defect in the 100-file benchmark represents an isolated Python defect. Crucially, the agent is evaluated against a hidden oracle test that the model never sees during generation or reflection.

The benchmark was executed with temperature: 0.0 and repeat_penalty: 1.05 on an RTX 3050 Laptop GPU (plugged into AC power):

Metric Single-Shot (Pass@1) Reflection Loop (Pass@overall)
Defects Repaired 38 / 100 (38%) 58 / 100 (58%)
Pass Rate Delta Baseline +20% absolute gain
Avg Repair Latency 29.6s (unoptimized) 7.80s (optimized)
Peak VRAM Consumption 2,532 MB 2,532 MB

The multi-turn reflection loop recovered 20 defects that failed on the first attempt. When provided with raw pytest traceback frames and compiler diagnostics, the 3B model successfully reformulated its reasoning in one out of every three initial failures.

Furthermore, by caching static analysis results and pre-warming the Ollama context window, per-defect latency dropped by 74%—from nearly 30 seconds down to 7.80 seconds per file.

3. Where 3B Models Excel vs. Where They Fail

The benchmark data split cleanly across two distinct categories of bugs:

The 100% Pass@overall Domains (Syntactic & Local Logic)

Eight fault domains achieved a perfect 100% resolution rate across retries:

  • Mutable Default Arguments: def append_to(item, target=[])
  • Resource Leaks: Missing with open(...) context managers
  • Iteration Mutation: Mutating a list or dictionary while actively iterating over it
  • Identity vs. Equality: Misusing is instead of == for integers or string literals
  • Bitwise vs. Logical Operators: Confusing & with and in boolean evaluations
  • Sort-in-Place Returns: Assigning result = items.sort() (which returns None)
  • Unhandled NoneType: Omitting guard checks prior to attribute lookups

In these domains, the bug is visible directly in the local code structure. The 3B model recognized the anti-pattern immediately.

The 0% Pass@overall Domains (Runtime Semantic Blind Spots)

Six categories scored 0% across every single defect and every single reflection retry:

  • Float Precision Comparisons: 0.1 + 0.2 == 0.3 without math.isclose()
  • Generator Exhaustion: Consuming an iterator twice without re-initialization
  • String .rstrip() Misuse: Using rstrip(".com") as a suffix remover
  • Unhashable Collections in Sets: Inserting dictionary references into sets
  • Off-by-One Boundary Slices: Zero-indexing fencepost errors

These bugs share a critical characteristic: their syntax looks completely clean. A model that operates primarily via token pattern matching cannot detect the flaw by inspecting token relationships alone—it must simulate the Python runtime VM in its latent space, where 3B parameters fall short.

4. The 67% Confirmation Bias Problem

The most revealing discovery in the benchmark emerged from analyzing the 42 failed defects:

In 67% of failed cases, the model’s self-generated test suite PASSED on its faulty patch, but FAILED the hidden oracle test.

When an autonomous agent is tasked with writing both the patch and the verification tests without an external anchor, it exhibits severe confirmation bias. The model writes tests that test its own assumptions rather than the actual specification:

# The Model's Buggy Patch:
def clean_domain(url: str) -> str:
    return url.rstrip(".com")

# The Model's Self-Generated Test (which PASSED):
def test_clean_domain():
    assert clean_domain("example.com") == "example"  # Passes!

# The Hidden Oracle Test (which FAILED):
def test_oracle():
    assert clean_domain("telecom.com") == "telecom"  # Fails! Returns "tele"

Because url.rstrip(".com") stripped all trailing characters matching {'c', 'o', 'm', '.'}, "telecom.com" became "tele". The model’s own unit test passed, the agent declared victory, and the audit report closed as “fixed.”

Takeaways for Agentic AI Engineers

  1. Local SLMs are Viable for Developer Tooling: A 4-bit 3B model running under 2.5GB VRAM can reliably remediate over 50% of routine code defects in under 8 seconds.
  2. Reflection Loops Matter: Agentic reflection provided a +20% absolute repair boost over raw zero-shot generation.
  3. Never Trust Agent Self-Verification: Self-written tests are susceptible to confirmation bias. Real-world autonomous agents must validate against immutable external invariants, AST linters, and independent oracle suites.

The complete benchmark harness, dataset, and LangGraph agent are open-source on GitHub.