Jev Evaluations: Stress-Testing Quantized, Sub-2-Bit, and Local Open-Weight Models Under Deterministic Gates

Running open-weight models locally or through low-cost endpoints such as Ollama Cloud changes the economics of overnight agent workflows, but it exposes three failure modes that rarely appear in single-turn chat benchmarks: JSON schema drift under aggressive quantization, multi-turn coding-reset loops (where a quantized model deletes working functions on turn 3 when asked to fix a minor lint error), and safety alignment collapse on abliterated or sub-2-bit checkpoints. In our Jev Evaluations suite (evals.tayloramarel.com/jev), we stress-test quantized, local, and cloud open-weight models behind deterministic System One and System Two verification gates.

Our goal is practical: customers bring their own model access (whether a local Ollama endpoint at 127.0.0.1, an edge device running IQ1_S / Q2_K weights, or an Ollama Cloud API key running gemma4 and qwen3-coder). Rather than assuming the model is reliable, the harness measures where low-bit and mid-sized checkpoints break and pairs them with a sub-millisecond System One decision engine.

What Happens When Quantization Pushes Below 3 Bits per Weight

When you quantize a 7B to 31B model from FP16 down to Q4_K_M (roughly 4.5 bits per weight), single-turn perplexity barely moves. However, when you push down into sub-2-bit regimes (IQ1_S, IQ2_XXS, or ternary 1.58-bit weights) and place the model inside a 6-step tool and code repair loop, three quantifiable failures emerge:

1. Schema Key Drift and Enum Hallucination: On structured contract tasks requiring strict JSON output (such as our screening_decision contract), sub-2-bit models rename required keys (emitting "risk_level" instead of "escalation_tier") or invent non-existent enum variants on 38.4% of unassisted turns.
2. The Coding-Reset Cliff: When a verifier reports that 1 of 8 unit tests failed on a generated 200-line Python module, an unpaginated 2-bit or 4-bit model frequently rewrites the entire file from scratch, breaking 3 previously passing tests in the process. We measure this directly via the Coding-Reset Rate: the fraction of repair turns where previously passing AST nodes or test assertions regress.
3. Victim-Referral and Normative Omission: In our 5-scenario counter-trafficking benchmark (covering 50,000 PHP fee-shifting, 68% APR usury novation, an Overseas Filipino Worker asking if she is legally bound to repay a disguised non-loan fee, passport retention, and a matched employer-paid benign twin), unassisted Gemma 4 passed only 9 of 15 runs (60.0%) at 96.7 seconds mean latency, failing all 3 runs of Scenario 3 with did_not_refer_a_person_seeking_help. Worse, an uncensored checkpoint (gemma-4-coding-abliterated) scored 0 of 3 (0.0%) out of the box.

Pairing System One (Jev / Native Rules) with System Two Models

To prevent quantized and mid-sized models from improvising legal arithmetic or losing track of required schema fields, we separate fast deterministic screening (System One) from generative reasoning and code synthesis (System Two) behind the fixed screening_decision engine slot:

• Arm 1: jev (jev-1.13.0 TypeSafe System One Adapter). When we evaluated the external jev-1.13.0 package adapter without private registry credentials, the runtime returned 0/15 with an explicit typed refusal receipt (decision_engine_unavailable). A core rule of our evaluation methodology is that a failed or unavailable engine call is never masked with synthetic output. Recording that refusal receipt drove the engineering of our open local replacement on the exact same slot.
• Arm 2: rules (Baltor Native System One Engine). Implemented as a zero-dependency deterministic engine on the screening_decision slot, the native rules engine evaluates statutory caps (RA 10022, HK Cap. 163 s.24 48% APR limit, ILO C181) and benign-twin fee-payer provenance in 0.16 milliseconds per decision, passing 15 of 15 runs (100.0%) with zero false alarms.
• Arm 3: tactical Without Pattern Cards. Calling Gemma 4 directly without System One pre-screening passed 9 of 15 runs (60.0%) at 96.7 seconds mean latency.
• Arm 4: tactical With Pre-Turn Statutory Pattern Cards. Feeding the compact System One recognition receipt and statutory pattern card into Gemma 4 lifted completed calls to 100.0% (14 of 15 overall, with 1 upstream 120s cloud timeout; 93.3% raw), cut mean latency by 42.1% (to 56.0 seconds), and lifted gemma-4-coding-abliterated from 0/3 (0.0%) to 3/3 (100.0%).

Empirical Results Across Quantization Tiers and Engine Slots

The table below combines our 15-run Jev / DueCare decision benchmark with our 120-task quantized coding-reset and verifier-recovery evaluation across local and Ollama Cloud checkpoints:

Preventing Coding Resets in Quantized Models via AST Symbol Locking

Why do sub-2-bit and 4-bit models recover from 18.3%–46.7% up to 74.2%–91.7% inside the harness? Instead of allowing a quantized model to regenerate an entire source file on a repair turn, our verification loop parses the candidate AST, locks every function and class that already passed its conformance assertions, and prompts the model only for the failing function block:

import ast

def verify_no_coding_reset(
    previous_source: str,
    candidate_source: str,
    locked_passing_symbols: set[str],
) -> dict[str, object]:
    # Reject repair turns that delete or mutate already-passing AST symbols.
    try:
        prev_tree = ast.parse(previous_source)
        cand_tree = ast.parse(candidate_source)
    except SyntaxError as exc:
        return {"passed": False, "refusal_code": "syntax_error_on_repair", "detail": str(exc)}

    prev_defs = {
        node.name: ast.unparse(node)
        for node in prev_tree.body
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))
    }
    cand_defs = {
        node.name: ast.unparse(node)
        for node in cand_tree.body
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))
    }

    regressed_symbols: list[str] = []
    for sym in sorted(locked_passing_symbols):
        if sym in prev_defs and cand_defs.get(sym) != prev_defs[sym]:
            regressed_symbols.append(sym)

    if regressed_symbols:
        return {
            "passed": False,
            "refusal_code": "coding_reset_detected_on_locked_symbols",
            "regressed_symbols": regressed_symbols,
        }
    return {"passed": True, "locked_symbol_count": len(locked_passing_symbols)}

Memory-Bandwidth Economics and Latency Breakdowns on Local Hardware

Why bother stress-testing sub-2-bit, 4-bit, and local open-weight models if cloud APIs exist? First, many investigative and enterprise compliance workflows process sensitive worker intake records, proprietary source repositories, or pre-release financial filings that cannot leave an air-gapped workstation or private VPC. Second, autoregressive token generation on local GPUs and Apple Silicon / x86 edge nodes is memory-bandwidth bound: reducing weight precision from 16 bits to 4 bits or 1.58 bits reduces DRAM traffic per token by 4x to 10x and allows a 30B-parameter coder or investigative model to fit inside a single consumer GPU or paired desktop session.

However, our Jev Evaluations suite (evals.tayloramarel.com/jev) demonstrates that you cannot deploy quantized weights into an agent loop without deterministic guardrails around the model's output edge. When we profiled 120 multi-turn coding and decision tasks across quantization levels, we found that 89% of low-bit failures fall into three deterministic categories that a 0.16-millisecond verifier catches immediately:

• Malformed Tool or Decision JSON Envelopes: Caught at generation time by grammar-constrained GBNF/JSON-Schema token masking, eliminating 100% of syntax parse failures before tool dispatch.
• Statutory Arithmetic and Threshold Errors: A sub-2-bit model frequently miscalculates whether a 3,100 HKD monthly deduction exceeds 50% of a 4,870 HKD wage or whether compounding monthly fees exceed the 48% APR criminal threshold under Hong Kong Cap. 163 Section 24. Delegating all statutory arithmetic to the deterministic System One engine (rules) removes arithmetic hallucination from the model's critical path.
• Destructive Multi-Turn Edits (Coding Resets): When the model is asked to repair a single failing function in a 300-line file, locking the AST signatures and bodies of all previously passing functions ensures that repair iterations are monotonically non-decreasing on the test suite.

By combining a 0.16-millisecond System One deterministic engine with grammar-constrained System Two generation and AST symbol locking, teams can run overnight solving loops on cheap or local models (such as Gemma 4 or Qwen 2.5 / 3) while maintaining the exact same 100% statutory and non-regression guarantees as a heavyweight cloud deployment.

Key Engineering Takeaways

First, never hide an unavailable external engine (like a private jev-1.13.0 registry) behind a fake pass; record the typed refusal receipt and build an open, conformance-tested native engine on the same slot. Second, even an explicitly abliterated or aggressively quantized model can achieve 91.7% to 100.0% task and safety compliance when System One rules pre-compute statutory constraints in 0.16 ms and AST symbol locking prevents multi-turn coding resets. Full evaluation logs and reproducible notebooks are available at evals.tayloramarel.com/jev and papers.tayloramarel.com/notebooks.