Every production LLM system has retries. And nearly every one of them is broken in the same way. The pattern is familiar: your language model produces output that fails validation. Maybe it hallucinated a forbidden term, exceeded a length constraint, or violated a format requirement. So you retry. Same prompt, same constraints, same prayer to the RNGeesus that this time the dice will roll your way. Sometimes it works. Often it doesn’t. And when it doesn’t, you’ve burned tokens, added latency, and still ended up in your fallback path. The problem isn’t that you’re retrying. The problem is that you’re retrying wrong. What is needed is something I call constraint escalation.
The Naive Retry Trap
Consider a typical constraint violation scenario. You’ve instructed your model not to reveal certain information—let’s say the word “secret” in a game narrative context. The model generates output containing “secret.” Validation catches it. You retry.
But here’s the thing: the model doesn’t know it failed. It doesn’t know why it failed. You’re feeding it the exact same prompt with the exact same constraints, hoping that stochastic sampling will produce a different result. Ralphing is not the solution.
This is the equivalent of asking someone to guess a number, telling them they’re wrong, and then asking them to guess again without telling them whether they were too high or too low. Sometimes you get lucky on retry two or three. But you’re not improving your odds with each attempt—you’re just rolling the same weighted dice repeatedly.
Constraint Escalation: The Feedback Loop
The insight behind constraint escalation is simple: each failure gives you information. Use it.
When validation fails, you don’t just know that the output was wrong—you know specifically what was wrong. The word “secret” appeared in position 47. A length limit was exceeded by 23 characters. A required field was missing entirely.
This information is gold. Instead of discarding it and hoping for better luck, you can use it to construct new, more specific constraints for the retry attempt.
var violation = new ConstraintViolation(
constraint: originalConstraint,
violatingText: "secret"
);var escalatedConstraints = retryPolicy.GenerateRetryConstraints(
violations: new[] { violation },
attemptNumber: 1
);The result isn’t just a retry—it’s a targeted retry with additional constraints derived directly from the failure mode.
Escalation Strategies
There are several ways to generate escalated constraints from violations. In practice, I’ve found two strategies that cover most cases:
Specific Prohibition. When the model produces forbidden content, extract the exact violating text and add an explicit prohibition. If the original constraint was “do not reveal secrets” and the model said “secret,” the escalated constraint becomes “do not say or imply: ‘secret’.” This transforms an abstract rule into a concrete prohibition.
Requirement Hardening. Not all constraints are created equal. Soft requirements (“try to keep responses concise”) have wiggle room that models will exploit. When a soft requirement is violated, escalate it to a hard requirement (“responses MUST be under 100 words”). The escalated constraint uses stronger language and moves from suggestion to mandate.
Combining both strategies gives you full escalation mode: each retry becomes progressively more constrained, more specific, and more likely to succeed.
The Math of Constraint Escalation
Let’s think about this probabilistically. Suppose your base prompt has a 70% chance of producing valid output. With naive retries, each attempt maintains that same 70% success rate regardless of previous failures. After three attempts, your cumulative success rate reaches 97.3%. Not bad, but you’re paying for three full inference calls in the worst case.
Now consider escalation. Each failed attempt improves the next attempt’s odds by adding specific constraints. If we conservatively estimate a 10% improvement per escalation, your first attempt succeeds 70% of the time, your second attempt (with escalated constraints) succeeds 80% of the time, and your third attempt (with doubly-escalated constraints) succeeds 90% of the time. After three attempts, your cumulative success rate climbs to 99.4%. More importantly, you reach that success rate with fewer expected attempts on average because each retry is more likely to hit than the last.
Implementation Considerations
Constraint escalation isn’t free. You need infrastructure to support it.
Your validation layer must return structured violation data, not just pass/fail. You need to know what failed and ideally capture the exact violating content so you can reference it in escalated constraints.
You also need logic to transform violations into new constraints. This can be rule-based, as shown above, or even model-assisted for more complex constraint types that require natural language reformulation.
Your prompt system must support dynamic constraint injection as well. Hard coded prompt strings won’t cut it when you need to append violation-specific prohibitions at runtime.
Finally, you should set escalation limits. At some point, adding more constraints hits diminishing returns or even becomes counterproductive, since over-constrained prompts can and will confuse models. Define a maximum escalation level and fall back gracefully when you exceed it.
The Bigger Picture
Constraint escalation is one piece of what I call a “governance control plane” approach to LLM systems. The core philosophy is simple: don’t trust the model, but don’t fight it either. Instead, create feedback loops that guide stochastic outputs toward deterministic guarantees.
The model remains probabilistic, and that’s where its power comes from. But the machine around it becomes increasingly deterministic with each retry, each escalation, and each piece of information extracted from failures.
This is how you ship LLM-powered features that actually work in production. You don’t hope the model behaves. You build systems that adapt when it doesn’t. This pattern is implemented in LlamaBrain, an open-source deterministic AI governance control plane. The escalation logic lives in RetryPolicy.cs for those who want to see the full implementation.