Beyond p^n: The Mathematics of Agentic Reliability and Harness Engineering

The transition from a linear Retrieval-Augmented Generation (RAG) pipeline to an agentic architecture is often framed as an upgrade in "intelligence." But under the hood, it is fundamentally an upgrade in probability.

If you have built a naive, multi-step LLM pipeline, you have likely encountered cascading failure. This post explores the exact mathematics of why linear pipelines collapse, how agentic retry loops fix them, and why your system is ultimately only as strong as your evaluation harness.

1. The Open-Loop Trap: The Math of "AND"

In a standard, fixed pipeline (e.g., Retrieve -> Rerank -> Synthesize -> Format), the architecture is an open loop. To get a successful final output, the LLM must succeed at Step 1, AND Step 2, AND Step 3.

If your underlying model has a zero-shot accuracy of p for any given task, the probability of successfully navigating an n-step pipeline is the product of those probabilities:

$$P(\text{success}) = p^n$$

Because p is a fraction, p^n decays exponentially. If your model gets it right 80% of the time (p = 0.8), a 4-step pipeline has a success rate of just 40.9%. The system is mathematically guaranteed to degrade as it scales.

2. The Agentic Loop: The Math of "OR"

Agentic architectures replace the "AND" with an "OR" by introducing a feedback loop. Instead of blindly passing data to the next node, an agent evaluates the result. If the result is flawed, it tries again, up to a limit of k retries.

Assuming the system can perfectly detect when an error occurs, the probability of failing a step is no longer (1-p). It is the probability of failing every single retry: (1-p)^k.

Subtracting this from 100% gives us the probability of succeeding at least once during those k attempts:

$$P(\text{step success}) = 1 - (1-p)^k$$

With p = 0.8 and k = 3, the success rate for a single step jumps from 80% to 99.2%. When you multiply these fortified nodes together, the pipeline stabilizes.

3. The Evaluator Bottleneck: Enter q

The previous equation makes a massive, often fatal assumption: that the agent perfectly recognizes its own failures. In reality, the evaluation step has its own probability of correctness, which we call q. When the agent generates an answer, the evaluator might falsely accept a hallucination (False Positive) or falsely reject a correct answer (False Negative).

When we account for an imperfect evaluator q, the math shifts from basic probability into a stochastic Markov chain. The master equation for a single agentic step becomes:

$$P(\text{step success}) = \left( \frac{p \cdot q}{p \cdot q + (1-p)(1-q)} \right) \left( 1 - (p(1-q) + (1-p)q)^k \right)$$

The False Positive Ceiling

Notice the fraction on the left side of the equation: $$\frac{p \cdot q}{p \cdot q + (1-p)(1-q)}$$.

This represents the absolute mathematical ceiling of your agent. Even if you give the agent infinite retries (k = infinity), it can never exceed this success rate. If both your generator and evaluator have an accuracy of 70% (p = 0.7, q = 0.7), your step's maximum possible success rate is capped at roughly 84%. False positives will eventually let bad data leak through the loop.

4. Harness Engineering: Breaking the Ceiling

To maximize the master equation, we must maximize q. This is the domain of Harness Engineering - the design of the environment that catches, tests, and evaluates the agent's output.

Harnesses generally fall into two categories:

Indeterministic Harness (LLM-as-a-Judge)

This relies on another prompt (or a separate model) to evaluate the output. "Does this summary look accurate?"

• The Problem: q remains fractional. You are fighting probability with probability.
• The Mitigation: If you must use an LLM judge, the task must be highly asymmetrical. Evaluating is easier than generating, so q is naturally higher than p. Furthermore, developers should route evaluations to heavier, highly capable models to push q as close to 0.99 as economically feasible.

Deterministic Harness (The Golden Rule)

The most robust agents do not use LLMs to check LLMs. They offload evaluation to strict, deterministic code.

• Did the Python script execute without throwing a traceback?
• Did the financial API return a 200 OK status with valid JSON?
• Does the generated output conform strictly to the required AST (Abstract Syntax Tree) schema?

By using code as the harness, q becomes 1.0.

When q = 1.0, the mathematical ceiling shatters. The left side of our master equation becomes 1, and the formula collapses back to the ideal 1 - (1-p)^k. A deterministic harness allows an agent to safely iterate until it hits a guaranteed success.

5. Human-in-the-Loop (HITL) as the Ultimate Harness

There are scenarios where a deterministic harness is impossible to build, and an indeterministic harness is too risky. If an agent is writing a sensitive client email or executing a live database mutation, a false positive is catastrophic.

This is where Human-in-the-Loop (HITL) architecture becomes a mathematical necessity.

HITL should not be viewed as a failure of automation; it is a dynamic routing strategy. When an agentic step involves high-stakes subjectivity, the retry loop routes the evaluation to a human operator.

1. The human acts as an evaluator with a functional q ≈ 1.0.
2. The human can either accept the output (Terminal Success), or reject it and provide targeted feedback.
3. The LLM processes the human feedback on the next iteration, artificially boosting its generation probability (p) for the subsequent attempt.

Conclusion

Building a production-ready agentic system is not about choosing a smarter foundational model. It is about actively managing the mathematical probabilities of your pipeline. By keeping steps short (n), leveraging retry loops (k), strictly enforcing deterministic harnesses wherever possible (q = 1.0), and utilizing HITL for high-risk evaluations, you can engineer a system that naturally self-corrects and vastly outperforms traditional linear RAG.