Showing posts with label human-in-the-loop. Show all posts
Showing posts with label human-in-the-loop. Show all posts

Monday, October 19, 2026

The Rise of Decision-Model-as-a-Judge

Automated AI evaluation has reached an architectural inflection point. For the past three years, engineering teams evaluating GenAI applications—from RAG pipelines to autonomous agent workflows—relied almost exclusively on LLM-as-a-Judge. Whenever an application needed to verify whether a response was grounded, whether a retrieved passage was relevant, or whether an agent’s chosen action was valid, the default design pattern was to prompt an expensive frontier model to generate a critique, followed by a final verdict.

That approach is buckling under production realities. Generative judges introduce seconds of latency, burn millions of unnecessary output tokens, and produce uncalibrated free-form text that brittle regular expressions must parse back into operational decisions.

To solve this, modern production stacks are shifting to the Decision-Model-as-a-Judge pattern: decoupling probabilistic evaluation from deterministic business policy by deploying purpose-built decision models.


1. The Ghost of Evaluations Past: Why logprobs and logit_bias Died

To understand why dedicated decision models emerged, we have to look at the collapse of early probability-extraction hacks.

Between 2020 and 2023, developers frequently evaluated outputs without generating full explanations by probing the model’s internal logits. The canonical technique was straightforward:

  1. Prompt a completion model with a yes/no question (e.g., "Is the answer supported by the text? Answer Yes or No:").
  2. Set max_tokens=1.
  3. Apply logit_bias to force the output token space exclusively toward the token IDs for Yes and No.
  4. Inspect top_logprobs to extract raw log-probabilities:

$$\log P(\text{Yes}) \quad \text{and} \quad \log P(\text{No})$$

  1. Normalize them via softmax to derive a synthetic "confidence score":

$$P(\text{Yes}) = \frac{e^{\text{logit}(\text{Yes})}}{e^{\text{logit}(\text{Yes})} + e^{\text{logit}(\text{No})}}$$

The Extinction Event

This mechanism died as generative architectures evolved along three vectors:

  • Tokenization Fragility and Calibration Drift: A single word like " Yes", "Yes", " yes", or "YES" occupies multiple distinct token IDs depending on leading whitespace and BPE splits. logit_bias frequently caused token collision artifacts, broke instruction-following, or triggered infinite sampling degeneration. More critically, raw softmax logits from generative models are notorious for poor calibration—a model could output 99% probability on a token simply due to frequency bias in pretraining corpora.
  • The Rise of Reasoning Models: Modern reasoning architectures utilize extensive internal chains of thought before emitting an answer. When a model computes thousands of hidden reasoning tokens, inspecting the logprob of the final terminal token ("Yes") is virtually meaningless. That probability is conditioned on the model's own generated rationalization, making it asymptotically approach 1.0. It reflects conditional certainty given its internal chain-of-thought, rather than genuine epistemic uncertainty over the supplied evidence.
  • API Redaction and Architectural Opacity: Frontier model providers increasingly treat raw logit distributions as sensitive IP or security vectors against model distillation. On reasoning endpoints, logprobs are often entirely disabled, restricted, or unrepresentative of intermediate verification states.

Engineering teams were left with a bad tradeoff: accept a multi-second, expensive LLM-as-a-Judge generating wordy explanations, or train and host dozens of brittle custom classification heads.


2. The Core Pattern: Probabilistic Judgment vs. Deterministic Policy

The Decision-Model-as-a-Judge pattern resolves this dilemma by separating judgment from policy.

Probabilistic Judgment (AI Model)
               +
Deterministic Decision Policy (Code / Rules)
               =
Operational Decision (Green / Amber / Red)

Instead of asking a general-purpose model to generate prose, the system queries a purpose-built Decision Model trained to evaluate bounded questions directly over input evidence.

The model outputs structured primitives—predicates, categorical choices, or ordinal rubric scores—alongside calibrated probability distributions. The application layer then evaluates those probabilities against business-defined risk thresholds.

       AI Application / Evaluation Pipeline
                         |
                         v
                Evaluation Evidence
       +------------------------------------+
       | AI Output, Retrieved Context,      |
       | Agent Trace, Source Documents      |
       +------------------------------------+
                         |
                         v
               Evaluation Definition
       (Questions / Rubrics / Allowed Choices)
                         |
                         v
             +-----------------------+
             |   Decision Model      |
             | (Jev, Decision-1, etc)|
             +-----------------------+
                         |
           +-------------+-------------+
           |             |             |
           v             v             v
       Decision        Score      Probability /
                                   Confidence
           +-------------+-------------+
                         |
                         v
               Deterministic Policy
            (Thresholds & Risk Rules)
                         |
           +-------------+-------------+
           |             |             |
           v             v             v
         GREEN         AMBER          RED
       (Accept)     (TBD/Review)   (Reject)
                         |
                  +------+------+
                  |             |
                  v             v
           LLM-as-a-Judge     HITL

3. The 2026 Decision-Model Ecosystem

A distinct class of non-generative, decision-oriented architectures now handles this layer:

Model / API Core Architecture & Paradigm Supported Primitives Native Outputs
TypeSafe Jev Purpose-built "System One" decision model; non-generative inference over state + typed queries Choice, Score, Noul Categorical probabilities, calibrated confidence intervals, 64K context
Microsoft-Decision-1 Post-trained Qwen-based decision engine optimized for workflow control & routing Yes/No, Multiple-choice, Ordinal ratings, Rubric scoring Calibrated probabilities over discrete candidate heads
OpenAI Decisions API Dedicated decision evaluation endpoint (powered by GPT-6 Luna) Predicate, Choice, Score Explicit probability distributions over defined criteria

Unlike traditional LLM inference, these models do not run autoregressive token generation loops to construct explanations. They evaluate the sequence representation across designated choice heads or classification layers, returning structured JSON containing probabilities and score distributions in a single pass.


4. Latency and Cost Arbitrage

Deploying Decision Models changes the operational economics of running evaluation in production:

Generative LLM-as-a-Judge:
[Input Tokens] ---> [Autoregressive Generation (300-800 tokens of reasoning)] ---> Final Score
Latency: 1,500ms - 5,000ms | High Cost (Generation tokens billed at 3x-4x input rate)

Decision-Model-as-a-Judge:
[Input Tokens + Choice Schema] ---> [Single-Pass Sequence Evaluation] ---> Probabilities
Latency: 80ms - 300ms      | Low Cost (Zero generation overhead)

The Cost Reality at Scale

Consider an application processing 500,000 evaluations per day across three dimensions (groundedness, relevance, safety):

  • Generative Judge (e.g., GPT / Claude): Prompting for a critique and JSON output generates an average of 400 tokens per evaluation. At 1.5M evaluations daily, that accounts for 600 million generated output tokens/day, generating thousands of dollars in daily evaluation spend alone.
  • Decision Model: Zero tokens are generated. Pricing is anchored on raw context processing or flat per-query decision units—typically reducing evaluation operational expenses by 80% to 92%.

Runtime Gating vs. Post-Mortem Logging

Because a decision model executes in sub-250ms timeframes, automated evaluations can shift from asynchronous offline log audits into synchronous runtime execution gates. An application can verify evidence grounding or check for policy violations inline before streaming the final response to an end user.


5. The Tri-State Policy: Green, Amber, Red

A critical anti-pattern in evaluation systems is treating an AI model's probability score directly as a binary Pass/Fail. A probability of 0.81 is a measurement of evidence, not an operational instruction.

Production pipelines require an intermediate state to account for epistemic uncertainty:

  • GREEN (Accept): Confidence meets or exceeds application acceptance requirements. Execution proceeds automatically.
  • AMBER (TBD / Escalate): The model cannot conclusively confirm or reject the criterion based on the evidence provided. The transaction is escalated.
  • RED (Reject): High confidence that the output violates the criteria, is ungrounded, or fails safety rubrics. The system triggers fallback, retry, or circuit breaking.
Metric: Groundedness P(Grounded)
0.00 -------------------- 0.70 ----------------------- 0.92 ----------------- 1.00
             RED                          AMBER                     GREEN
       [Direct Reject]           [Escalate: LLM / HITL]       [Direct Accept]

Escalating the Amber Zone

By isolating borderline cases, engineering teams can implement targeted escalation pipelines:

  1. Escalation to LLM-as-a-Judge: Amber cases (typically 5%–15% of total volume) can be routed to an expensive, frontier reasoning model tasked with full chain-of-thought analysis and natural language synthesis. You reserve expensive reasoning models for queries that genuinely require deliberation.
  2. Human-in-the-Loop (HITL) Queues: High-stakes decisions (e.g., contract validation, healthcare extraction) route directly to human reviewers, prioritizing operator attention where machine certainty is lowest.

6. Real-World Blueprints

Blueprint A: RAG Groundedness & Relevance Verification

In a retrieval-augmented system, evaluating groundedness requires assessing whether every factual claim in the generation is supported by the retrieved context.

# Normalized Decision Model Call
decision_result = decision_client.evaluate(
    model="typesafe/jev-1.13",
    evidence={
        "retrieved_context": retrieved_passages,
        "generated_answer": response_draft
    },
    question={
        "type": "choice",
        "prompt": "Is the generated answer entirely supported by the context without extrapolation?",
        "options": ["fully_grounded", "partially_grounded", "hallucinated"]
    }
)

# Probabilistic Output
# decision_result.probabilities = {"fully_grounded": 0.88, "partially_grounded": 0.09, "hallucinated": 0.03}

# Deterministic Application Policy
p_grounded = decision_result.probabilities["fully_grounded"]

if p_grounded >= 0.90:
    emit_response(response_draft)               # GREEN
elif p_grounded >= 0.70:
    escalate_to_reasoning_eval(response_draft)  # AMBER -> Deep CoT Judge
else:
    trigger_fallback_retrieval()                # RED -> Remediate

Blueprint B: Autonomous Agent Action Verification

In agentic architectures, execution loops need deterministic guardrails prior to tool invocation:

Agent Goal + Execution History + Selected Tool Action
                         |
                         v
   Decision-Model-as-a-Judge (Microsoft-Decision-1)
                         |
               P(Action Appropriate)
                         |
     +-------------------+-------------------+
     |                                       |
P >= 0.95 (Green)                      P < 0.95 (Amber/Red)
Execute Tool Call                      Block Tool & Request 
                                       Human-in-the-Loop Confirmation

Blueprint C: Document Extraction with Field-Level Triage

When extracting semi-structured entities from complex PDFs (e.g., invoices, tax returns, bills of lading), fields exhibit varying levels of ambiguity:

  • Invoice Number --> P(Correct) = 0.98 --> GREEN (Direct DB Write)
  • Vendor Address --> P(Correct) = 0.94 --> GREEN (Direct DB Write)
  • Payment Terms --> P(Correct) = 0.74 --> AMBER (Routed to Human Review UI)
  • Total Amount --> P(Correct) = 0.99 --> GREEN (Direct DB Write)

Human operators review only the ambiguous fields rather than validating entire documents from scratch, increasing operational throughput by orders of magnitude.


7. Implementation & Calibration Best Practices

1. Calibrate on Golden Datasets First

Never set Green/Amber thresholds based on intuition. Evaluate the decision model against a curated golden benchmark (e.g., 500 labeled examples). Calculate the Expected Calibration Error (ECE) and plot a reliability diagram to verify that a reported probability of 0.85 matches an empirical accuracy of ~85%.

2. Monitor Amber Rate Drift

Track the ratio of Green/Amber/Red classifications across rolling windows. If your Amber rate suddenly spikes from 8% to 26%, do not adjust your policy thresholds.

A surge in Amber indicates upstream distribution drift: the retrieval engine might be fetching lower-quality chunks, an upstream LLM update may have altered output formatting, or user traffic patterns may have shifted.

3. Decompose Complex Rubrics into Atomic Dimensions

Avoid asking omnibus questions such as: "Is this response polite, grounded, succinct, and structurally compliant?"

Decision models excel at bounded evaluation. Run parallel, atomic decision checks across each distinct dimension:

  • P(Grounded)
  • P(Compliant)
  • P(Complete)

Then, aggregate the resulting scores using deterministic business logic in code.


Summary Architecture Comparison

+---------------------------------------------------------------------------------+
|                                  Evaluation Stack                               |
+---------------------------------------------------------------------------------+
|  Tier 1: Deterministic Rules (Regex, JSON schema, blocklists)                   |
|  --> Cost: $0.00 | Latency: <5ms                                                |
+---------------------------------------------------------------------------------+
|  Tier 2: Decision-Model-as-a-Judge (Jev, Microsoft-Decision-1, OpenAI Decisions)|
|  --> Cost: ~$0.0001 - $0.0008 | Latency: 50-250ms                               |
|  --> Classifies: Green (Accept) / Red (Reject) / Amber (Escalate)               |
+---------------------------------------------------------------------------------+
|  Tier 3: Reasoning LLM-as-a-Judge (Frontier CoT Models)                         |
|  --> Cost: ~$0.01 - $0.05 | Latency: 2,000-6,000ms                              |
|  --> Handles: Complex synthesis, detailed root-cause diagnosis, Amber resolution|
+---------------------------------------------------------------------------------+
|  Tier 4: Human-in-the-Loop Review (HITL)                                        |
|  --> Resolves high-risk edge cases and informs golden dataset updates           |
+---------------------------------------------------------------------------------+

By transitioning bounded quality gates from generative LLMs to calibrated decision models, production systems achieve sub-second evaluation latency, drastically lower operational costs, and establish deterministic control over AI application risk.

Saturday, June 06, 2026

Beyond p^n: The Mathematics of Agentic Reliability and Harness Engineering

The transition from a linear Retrieval-Augmented Generation (RAG) pipeline to an agentic architecture is often framed as an upgrade in "intelligence." But under the hood, it is fundamentally an upgrade in probability.
If you have built a naive, multi-step LLM pipeline, you have likely encountered cascading failure. This post explores the exact mathematics of why linear pipelines collapse, how agentic retry loops fix them, and why your system is ultimately only as strong as your evaluation harness.

1. The Open-Loop Trap: The Math of "AND"

In a standard, fixed pipeline (e.g., Retrieve -> Rerank -> Synthesize -> Format), the architecture is an open loop. To get a successful final output, the LLM must succeed at Step 1, AND Step 2, AND Step 3.
If your underlying model has a zero-shot accuracy of p for any given task, the probability of successfully navigating an n-step pipeline is the product of those probabilities:
$$P(\text{success}) = p^n$$
Because p is a fraction, p^n decays exponentially. If your model gets it right 80% of the time (p = 0.8), a 4-step pipeline has a success rate of just 40.9%. The system is mathematically guaranteed to degrade as it scales.

2. The Agentic Loop: The Math of "OR"

Agentic architectures replace the "AND" with an "OR" by introducing a feedback loop. Instead of blindly passing data to the next node, an agent evaluates the result. If the result is flawed, it tries again, up to a limit of k retries.
Assuming the system can perfectly detect when an error occurs, the probability of failing a step is no longer (1-p). It is the probability of failing every single retry: (1-p)^k.
Subtracting this from 100% gives us the probability of succeeding at least once during those k attempts:
$$P(\text{step success}) = 1 - (1-p)^k$$
With p = 0.8 and k = 3, the success rate for a single step jumps from 80% to 99.2%. When you multiply these fortified nodes together, the pipeline stabilizes.

3. The Evaluator Bottleneck: Enter q

The previous equation makes a massive, often fatal assumption: that the agent perfectly recognizes its own failures. In reality, the evaluation step has its own probability of correctness, which we call q. When the agent generates an answer, the evaluator might falsely accept a hallucination (False Positive) or falsely reject a correct answer (False Negative).
When we account for an imperfect evaluator q, the math shifts from basic probability into a stochastic Markov chain. The master equation for a single agentic step becomes:
$$P(\text{step success}) = \left( \frac{p \cdot q}{p \cdot q + (1-p)(1-q)} \right) \left( 1 - (p(1-q) + (1-p)q)^k \right)$$

The False Positive Ceiling

Notice the fraction on the left side of the equation: $$\frac{p \cdot q}{p \cdot q + (1-p)(1-q)}$$.
This represents the absolute mathematical ceiling of your agent. Even if you give the agent infinite retries (k = infinity), it can never exceed this success rate. If both your generator and evaluator have an accuracy of 70% (p = 0.7, q = 0.7), your step's maximum possible success rate is capped at roughly 84%. False positives will eventually let bad data leak through the loop.

4. Harness Engineering: Breaking the Ceiling

To maximize the master equation, we must maximize q. This is the domain of Harness Engineering - the design of the environment that catches, tests, and evaluates the agent's output.
Harnesses generally fall into two categories:

Indeterministic Harness (LLM-as-a-Judge)

This relies on another prompt (or a separate model) to evaluate the output. "Does this summary look accurate?"
  • The Problem: q remains fractional. You are fighting probability with probability.
  • The Mitigation: If you must use an LLM judge, the task must be highly asymmetrical. Evaluating is easier than generating, so q is naturally higher than p. Furthermore, developers should route evaluations to heavier, highly capable models to push q as close to 0.99 as economically feasible.

Deterministic Harness (The Golden Rule)

The most robust agents do not use LLMs to check LLMs. They offload evaluation to strict, true/false code.
  • Format Checking: Did the model output pure, valid JSON, or did it break the code by adding conversational filler like "Here is your JSON:"?
  • Execution Checking: If the agent wrote a block of code, did it actually run successfully without crashing?
  • Rule Counting: If the prompt asked for exactly five product recommendations, does len(list) == 5?
By using code as the harness, q becomes 1.0.
When q = 1.0, the mathematical ceiling shatters. The left side of our master equation becomes 1, and the formula collapses back to the ideal 1 - (1-p)^k. A deterministic harness allows an agent to safely iterate until it hits a guaranteed success.

5. Human-in-the-Loop (HITL) as the Ultimate Harness

There are scenarios where a deterministic harness is impossible to build, and an indeterministic harness is too risky. If an agent is writing a sensitive client email or executing a live database mutation, a false positive is catastrophic.
This is where Human-in-the-Loop (HITL) architecture becomes a mathematical necessity.
HITL should not be viewed as a failure of automation; it is a dynamic routing strategy. When an agentic step involves high-stakes subjectivity, the retry loop routes the evaluation to a human operator.
  1. The human acts as an evaluator with a functional q ≈ 1.0.
  2. The human can either accept the output (Terminal Success), or reject it and provide targeted feedback.
  3. The LLM processes the human feedback on the next iteration, artificially boosting its generation probability (p) for the subsequent attempt.

Conclusion

Building a production-ready agentic system is not about choosing a smarter foundational model. It is about actively managing the mathematical probabilities of your pipeline. By keeping steps short (n), leveraging retry loops (k), strictly enforcing deterministic harnesses wherever possible (q = 1.0), and utilizing HITL for high-risk evaluations, you can engineer a system that naturally self-corrects and vastly outperforms traditional linear RAG.