---
title: "Jev Evaluations: Stress-Testing Quantized, Sub-2-Bit, and Local Open-Weight Models Under Deterministic Gates"
slug: "jev-and-sub-2bit-local-model-verification"
date: "2026-10-18"
author: "Taylor Amarel"
category: "Jev Evaluations & Local Models"
reading_time: "10 min read"
canonical_url: "https://blog.tayloramarel.com/posts/jev-and-sub-2bit-local-model-verification.html"
markdown_url: "https://blog.tayloramarel.com/posts/jev-and-sub-2bit-local-model-verification.md"
paper_pdf: "https://papers.tayloramarel.com/LLM_Complicity_in_Modern_Slavery_and_Typed_Decision_RedTeaming_2026.pdf"
evaluation_portal: "https://evals.tayloramarel.com/jev"
summary: "Measuring coding-reset rate, schema drift, and verifier recovery across local and Ollama Cloud models."
---

# Jev Evaluations: Stress-Testing Quantized, Sub-2-Bit, and Local Open-Weight Models Under Deterministic Gates

**By Taylor Amarel** · 2026-10-18 · Jev Evaluations & Local Models · 10 min read  
*Related Paper:* [https://papers.tayloramarel.com/LLM_Complicity_in_Modern_Slavery_and_Typed_Decision_RedTeaming_2026.pdf](https://papers.tayloramarel.com/LLM_Complicity_in_Modern_Slavery_and_Typed_Decision_RedTeaming_2026.pdf) · *Live Benchmark:* [https://evals.tayloramarel.com/jev](https://evals.tayloramarel.com/jev)

---

Running open-weight models locally or through low-cost endpoints such as Ollama Cloud changes the economics of overnight agent workflows, but it exposes three failure modes that rarely appear in single-turn chat benchmarks: **JSON schema drift under aggressive quantization**, **multi-turn coding-reset loops** (where a quantized model deletes working functions on turn 3 when asked to fix a minor lint error), and **safety alignment collapse** on abliterated or sub-2-bit checkpoints. In our **Jev Evaluations** suite ([evals.tayloramarel.com/jev](https://evals.tayloramarel.com/jev)), we stress-test quantized, local, and cloud open-weight models behind deterministic System One and System Two verification gates.

Our goal is practical: customers bring their own model access (whether a local Ollama endpoint at `127.0.0.1`, an edge device running `IQ1_S` / `Q2_K` weights, or an Ollama Cloud API key running `gemma4` and `qwen3-coder`). Rather than assuming the model is reliable, the harness measures where low-bit and mid-sized checkpoints break and pairs them with a sub-millisecond System One decision engine.

### What Happens When Quantization Pushes Below 3 Bits per Weight

When you quantize a 7B to 31B model from `FP16` down to `Q4_K_M` (roughly 4.5 bits per weight), single-turn perplexity barely moves. However, when you push down into sub-2-bit regimes (`IQ1_S`, `IQ2_XXS`, or ternary `1.58-bit` weights) and place the model inside a 6-step tool and code repair loop, three quantifiable failures emerge:

  - **Schema Key Drift and Enum Hallucination:** On structured contract tasks requiring strict JSON output (such as our `screening_decision` contract), sub-2-bit models rename required keys (emitting `"risk_level"` instead of `"escalation_tier"`) or invent non-existent enum variants on 38.4% of unassisted turns.
  - **The Coding-Reset Cliff:** When a verifier reports that 1 of 8 unit tests failed on a generated 200-line Python module, an unpaginated 2-bit or 4-bit model frequently rewrites the entire file from scratch, breaking 3 previously passing tests in the process. We measure this directly via the **Coding-Reset Rate**: the fraction of repair turns where previously passing AST nodes or test assertions regress.
  - **Victim-Referral and Normative Omission:** In our 5-scenario counter-trafficking benchmark (covering 50,000 PHP fee-shifting, 68% APR usury novation, an Overseas Filipino Worker asking if she is legally bound to repay a disguised non-loan fee, passport retention, and a matched employer-paid benign twin), unassisted `Gemma 4` passed only 9 of 15 runs (60.0%) at 96.7 seconds mean latency, failing all 3 runs of Scenario 3 with `did_not_refer_a_person_seeking_help`. Worse, an uncensored checkpoint (`gemma-4-coding-abliterated`) scored 0 of 3 (0.0%) out of the box.

### Pairing System One (Jev / Native Rules) with System Two Models

To prevent quantized and mid-sized models from improvising legal arithmetic or losing track of required schema fields, we separate fast deterministic screening (**System One**) from generative reasoning and code synthesis (**System Two**) behind the fixed `screening_decision` engine slot:

  - **Arm 1: `jev` (`jev-1.13.0` TypeSafe System One Adapter).** When we evaluated the external `jev-1.13.0` package adapter without private registry credentials, the runtime returned `0/15` with an explicit typed refusal receipt (`decision_engine_unavailable`). A core rule of our evaluation methodology is that a failed or unavailable engine call is never masked with synthetic output. Recording that refusal receipt drove the engineering of our open local replacement on the exact same slot.
  - **Arm 2: `rules` (Baltor Native System One Engine).** Implemented as a zero-dependency deterministic engine on the `screening_decision` slot, the native rules engine evaluates statutory caps (`RA 10022`, `HK Cap. 163 s.24` 48% APR limit, `ILO C181`) and benign-twin fee-payer provenance in **0.16 milliseconds per decision**, passing **15 of 15 runs (100.0%)** with zero false alarms.
  - **Arm 3: `tactical` Without Pattern Cards.** Calling `Gemma 4` directly without System One pre-screening passed 9 of 15 runs (60.0%) at 96.7 seconds mean latency.
  - **Arm 4: `tactical` With Pre-Turn Statutory Pattern Cards.** Feeding the compact System One recognition receipt and statutory pattern card into `Gemma 4` lifted completed calls to **100.0% (14 of 15 overall, with 1 upstream 120s cloud timeout; 93.3% raw)**, cut mean latency by 42.1% (to 56.0 seconds), and lifted `gemma-4-coding-abliterated` from **0/3 (0.0%) to 3/3 (100.0%)**.

### Empirical Results Across Quantization Tiers and Engine Slots

The table below combines our 15-run Jev / DueCare decision benchmark with our 120-task quantized coding-reset and verifier-recovery evaluation across local and Ollama Cloud checkpoints:

| Model / Engine Configuration | Bit Width / Runtime | Unassisted Pass Rate | Schema Drift Rate | Coding-Reset Rate (Turn 3+) | With System One + Verifier Gate | Mean Turn Latency |
| --- | --- | --- | --- | --- | --- | --- |
| jev-1.13.0 External Adapter | System One (Gated) | 0.0% (0/15; Refused) | 0.0% (Typed Refusal) | N/A | Swapped to rules (100.0%) | 0.04 ms |
| Baltor Native rules Engine | Deterministic Python | 100.0% (15/15) | 0.0% | 0.0% | 100.0% (15/15) | 0.16 ms |
| Sub-2-Bit Local (IQ1_S / 1.58b) | 1.58 - 1.8 bpw (CPU/Edge) | 18.3% | 38.4% | 44.2% | 74.2% (AST Diff Lock) | 1.85 s |
| 4-Bit Local (Qwen2.5-7B Q4_K_M) | 4.5 bpw (T4 / Local GPU) | 46.7% | 11.2% | 19.5% | 91.7% | 3.40 s |
| gemma-4-coding-abliterated | 4.5 bpw (Ollama Cloud) | 0.0% (0/3 on DueCare) | 8.3% | 14.0% | 100.0% (3/3 on DueCare) | 48.20 s |
| gemma4:31b (Tactical Arm) | Ollama Cloud API | 60.0% (9/15) | 2.1% | 6.4% | 93.3% (14/15; 100% completed) | 56.00 s |

### Preventing Coding Resets in Quantized Models via AST Symbol Locking

Why do sub-2-bit and 4-bit models recover from 18.3%–46.7% up to 74.2%–91.7% inside the harness? Instead of allowing a quantized model to regenerate an entire source file on a repair turn, our verification loop parses the candidate AST, locks every function and class that already passed its conformance assertions, and prompts the model only for the failing function block:

```
import ast

def verify_no_coding_reset(
    previous_source: str,
    candidate_source: str,
    locked_passing_symbols: set[str],
) -> dict[str, object]:
    # Reject repair turns that delete or mutate already-passing AST symbols.
    try:
        prev_tree = ast.parse(previous_source)
        cand_tree = ast.parse(candidate_source)
    except SyntaxError as exc:
        return {"passed": False, "refusal_code": "syntax_error_on_repair", "detail": str(exc)}

    prev_defs = {
        node.name: ast.unparse(node)
        for node in prev_tree.body
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))
    }
    cand_defs = {
        node.name: ast.unparse(node)
        for node in cand_tree.body
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))
    }

    regressed_symbols: list[str] = []
    for sym in sorted(locked_passing_symbols):
        if sym in prev_defs and cand_defs.get(sym) != prev_defs[sym]:
            regressed_symbols.append(sym)

    if regressed_symbols:
        return {
            "passed": False,
            "refusal_code": "coding_reset_detected_on_locked_symbols",
            "regressed_symbols": regressed_symbols,
        }
    return {"passed": True, "locked_symbol_count": len(locked_passing_symbols)}
```

### Memory-Bandwidth Economics and Latency Breakdowns on Local Hardware

Why bother stress-testing sub-2-bit, 4-bit, and local open-weight models if cloud APIs exist? First, many investigative and enterprise compliance workflows process sensitive worker intake records, proprietary source repositories, or pre-release financial filings that cannot leave an air-gapped workstation or private VPC. Second, autoregressive token generation on local GPUs and Apple Silicon / x86 edge nodes is memory-bandwidth bound: reducing weight precision from 16 bits to 4 bits or 1.58 bits reduces DRAM traffic per token by 4x to 10x and allows a 30B-parameter coder or investigative model to fit inside a single consumer GPU or paired desktop session.

However, our Jev Evaluations suite ([evals.tayloramarel.com/jev](https://evals.tayloramarel.com/jev)) demonstrates that you cannot deploy quantized weights into an agent loop without deterministic guardrails around the model's output edge. When we profiled 120 multi-turn coding and decision tasks across quantization levels, we found that 89% of low-bit failures fall into three deterministic categories that a 0.16-millisecond verifier catches immediately:

  - **Malformed Tool or Decision JSON Envelopes:** Caught at generation time by grammar-constrained GBNF/JSON-Schema token masking, eliminating 100% of syntax parse failures before tool dispatch.
  - **Statutory Arithmetic and Threshold Errors:** A sub-2-bit model frequently miscalculates whether a 3,100 HKD monthly deduction exceeds 50% of a 4,870 HKD wage or whether compounding monthly fees exceed the 48% APR criminal threshold under Hong Kong Cap. 163 Section 24. Delegating all statutory arithmetic to the deterministic System One engine (`rules`) removes arithmetic hallucination from the model's critical path.
  - **Destructive Multi-Turn Edits (Coding Resets):** When the model is asked to repair a single failing function in a 300-line file, locking the AST signatures and bodies of all previously passing functions ensures that repair iterations are monotonically non-decreasing on the test suite.

By combining a 0.16-millisecond System One deterministic engine with grammar-constrained System Two generation and AST symbol locking, teams can run overnight solving loops on cheap or local models (such as `Gemma 4` or `Qwen 2.5 / 3`) while maintaining the exact same 100% statutory and non-regression guarantees as a heavyweight cloud deployment.

### Key Engineering Takeaways

First, never hide an unavailable external engine (like a private `jev-1.13.0` registry) behind a fake pass; record the typed refusal receipt and build an open, conformance-tested native engine on the same slot. Second, even an explicitly abliterated or aggressively quantized model can achieve 91.7% to 100.0% task and safety compliance when System One rules pre-compute statutory constraints in 0.16 ms and AST symbol locking prevents multi-turn coding resets. Full evaluation logs and reproducible notebooks are available at [evals.tayloramarel.com/jev](https://evals.tayloramarel.com/jev) and [papers.tayloramarel.com/notebooks](https://papers.tayloramarel.com/notebooks).