A RAG agent returns a confident, well-written paragraph that is not in any of the passages you retrieved. The sources were right there in the prompt, and the model wrote around them anyway. You cannot ship that to a user, and you cannot read every answer yourself.
The usual fix is a second LLM call that grades the first. It costs another full generation, adds seconds, and fails in the same places the first one did — it is the same family of model making the same kind of mistake. A separate, typed, fast verifier is a better tool for the job.
This is the layer that sits after the model. For screening what goes in, see input guardrails and moderation for Laravel AI SDK agents — the two complement each other and this article assumes you already have the input side.
Generate the draft answer with the Laravel AI SDK#
Start with an ordinary agent. The only thing the guardrail needs from it is the answer text and the passages that were supposed to support it, so keep both in scope rather than throwing the retrieved context away after the prompt.
<?php
declare(strict_types=1);
namespace App\Ai\Agents;
use Laravel\Ai\Contracts\Agent;
use Laravel\Ai\Promptable;
final class DocsAnswerer implements Agent
{
use Promptable;
public function instructions(): string
{
return 'Answer the question using only the supplied passages. '
.'If the passages do not contain the answer, say you do not know.';
}
}
The calling service holds onto the passages. If you are retrieving with pgvector, this is the same $passages collection that came out of the similarity search in the Laravel AI SDK RAG pipeline.
$passages = $this->retriever->search($question, limit: 6);
$answer = (string) (new DocsAnswerer)->prompt(
"Question: {$question}\n\nPassages:\n".$passages->implode("\n---\n"),
);
Build the verification questions#
The Laravel AI SDK ships a Classification API backed by TypeSafe — Jev is a first-class provider, listed alongside the text and embedding labs. That means no thin HTTP wrapper over POST /v1/systemone, which is how I did it in confidence-gated ticket triage with Jev before the SDK covered it.
Set the key in .env:
TYPESAFE_API_KEY=
Three questions, asked together against the same state. Groundedness and citation support are Boolean questions, because the useful output is a probability you threshold. Policy is a Choice, because the outcomes are a closed set with different handling.
<?php
declare(strict_types=1);
namespace App\Ai\Verification;
use Laravel\Ai\Classification\Boolean;
use Laravel\Ai\Classification\Choice;
final class VerificationQuestions
{
/**
* @return array<string, Boolean|Choice>
*/
public static function all(): array
{
return [
'grounded' => new Boolean(
'Is every factual claim in `answer` supported by the text in `passages`?',
[
'true' => 'Every claim traces back to something stated in the passages.',
'false' => 'At least one claim is absent from, or goes beyond, the passages.',
],
),
'cites_correctly' => new Boolean(
'Does every passage number cited in `answer` actually contain the claim it is attached to?',
[
'true' => 'Each citation points at a passage that states the claim.',
'false' => 'At least one citation points at a passage that does not state the claim.',
],
),
'policy' => new Choice(
'How should this answer be handled before it is shown to a user?',
[
'ok' => 'Ordinary, in-scope answer with nothing that needs qualifying.',
'needs_disclaimer' => 'In scope, but touches legal, medical or financial territory.',
'refuse' => 'Out of scope, or gives advice the product must not give.',
],
),
];
}
}
Write the criteria as full sentences. Jev is reading them as the rubric, and "at least one claim is absent from, or goes beyond, the passages" draws a line that "not grounded" does not.
Send the answer and sources to Jev#
Pass structured state rather than a concatenated blob. Classification::of() accepts an array, and naming the fields lets the question instructions point at them with backticks the way they do above.
<?php
declare(strict_types=1);
namespace App\Ai\Verification;
use Illuminate\Support\Collection;
use Illuminate\Support\Facades\Log;
use Laravel\Ai\Classification;
final class AnswerVerifier
{
public function verify(string $question, string $answer, Collection $passages): Verdict
{
$result = Classification::of([
'question' => $question,
'answer' => $answer,
'passages' => $passages->values()->all(),
])->questions(VerificationQuestions::all())->classify();
Log::info('answer.verification.usage', [
'usage' => $result->usage,
'provider' => $result->meta->provider,
]);
return new Verdict(
grounded: $result['grounded']->probability,
citesCorrectly: $result['cites_correctly']->probability,
policy: $result['policy']->choice,
policyConfidence: $result['policy']->confidence,
);
}
}
One request carries all three questions, and output tokens are free, so a fourth question costs you almost nothing. Log $result->usage from day one — it is the only honest answer to "what is this guardrail costing us", and it slots into whatever you already have from tracking Laravel AI SDK token usage and cost.
Block, retry or escalate based on the result#
Jev supplies the assessment. Your application owns the decision, and that decision belongs in code you can read, test and change without retraining anything.
<?php
declare(strict_types=1);
namespace App\Ai\Verification;
enum Action
{
case Pass;
case Regenerate;
case Disclaim;
case Refuse;
}
final readonly class Verdict
{
public function __construct(
public float $grounded,
public float $citesCorrectly,
public string $policy,
public ?float $policyConfidence,
) {}
public function action(): Action
{
return match (true) {
// A confident refusal wins over everything else.
$this->policy === 'refuse' && ($this->policyConfidence ?? 0.0) >= 0.8 => Action::Refuse,
$this->grounded < 0.50 => Action::Refuse,
$this->grounded < 0.85 || $this->citesCorrectly < 0.85 => Action::Regenerate,
$this->policy === 'needs_disclaimer' => Action::Disclaim,
default => Action::Pass,
};
}
}
Three bands, not two. An answer at 0.60 groundedness is usually one that drifted in a sentence or two and will come back clean on a second attempt with the failure quoted back at it; an answer at 0.20 invented the whole thing and retrying wastes money. Regenerate once — a loop here is how you turn a 2-second request into a 20-second one.
$verdict = $this->verifier->verify($question, $answer, $passages);
$answer = match ($verdict->action()) {
Action::Pass => $answer,
Action::Disclaim => $this->withDisclaimer($answer),
Action::Regenerate => $this->regenerateOnce($question, $passages, $verdict),
Action::Refuse => $this->fallback(),
};
I start the thresholds high — 0.85 — and lower them once there is a week of logged verdicts to look at. Starting permissive and tightening later means the bad answers ship first.
Wrap it as reusable agent middleware#
Doing this at one call site is fine. Doing it at nine is how one of them quietly skips the check. Agent middleware moves it to the agent definition, so every prompt through that agent is verified whether the caller remembered or not.
php artisan make:agent-middleware VerifyGroundedness
The middleware needs the question and the passages, which the prompt string alone will not give it back cleanly. Push them into Laravel's Context at retrieval time — it is request-scoped, survives into queued agent runs, and lands in your logs for free, as covered in request-scoped logging with Context.
use Illuminate\Support\Facades\Context;
Context::add('rag.question', $question);
Context::add('rag.passages', $passages->values()->all());
Middleware runs once per generation step, so guard on isFinalStep — verifying an intermediate tool-calling step wastes a request on text the user will never see.
<?php
declare(strict_types=1);
namespace App\Ai\Middleware;
use App\Ai\Verification\Action;
use App\Ai\Verification\AnswerVerifier;
use App\Exceptions\UnverifiedAnswerException;
use Closure;
use Illuminate\Support\Facades\Context;
use Illuminate\Support\Facades\Log;
use Laravel\Ai\Gateway\StepResponse;
use Laravel\Ai\PendingStep;
final class VerifyGroundedness
{
public function __construct(private AnswerVerifier $verifier) {}
public function handle(PendingStep $step, Closure $next)
{
return $next($step)->then(function (StepResponse $response) use ($step) {
if (! $step->isFinalStep) {
return;
}
$verdict = $this->verifier->verify(
question: Context::get('rag.question', ''),
answer: $response->text,
passages: collect(Context::get('rag.passages', [])),
);
Log::info('answer.verified', [
'grounded' => $verdict->grounded,
'action' => $verdict->action()->name,
]);
if ($verdict->action() === Action::Refuse) {
throw new UnverifiedAnswerException($verdict);
}
});
}
}
Register it on the agent:
use Laravel\Ai\Contracts\HasMiddleware;
final class DocsAnswerer implements Agent, HasMiddleware
{
use Promptable;
public function middleware(): array
{
return [app(VerifyGroundedness::class)];
}
}
Two honest caveats. Middleware must return the result of $next, or its own StepResponse; anything else throws a LogicException. And rewriting the answer text from inside then() is not part of the documented surface, so the middleware raises UnverifiedAnswerException and the caller decides what the user sees. That keeps the substitution — fallback copy, disclaimer wrapper, support hand-off — in the layer that knows which it should be.
Test the guardrail with fakes#
Classification::fake() takes the API out of the loop entirely, so the decision table can be tested exhaustively and in milliseconds. Questions you omit still get a generated answer, so each case only states the number it is about.
<?php
use App\Ai\Verification\Action;
use App\Ai\Verification\AnswerVerifier;
use Laravel\Ai\Classification;
use Laravel\Ai\Prompts\ClassificationPrompt;
use Laravel\Ai\Responses\Data\BooleanAnswer;
use Laravel\Ai\Responses\Data\ChoiceAnswer;
it('maps groundedness probabilities onto actions', function (float $grounded, Action $expected) {
Classification::fake([[
'grounded' => new BooleanAnswer($grounded),
'cites_correctly' => new BooleanAnswer(0.99),
'policy' => new ChoiceAnswer('ok', [
'ok' => 0.97, 'needs_disclaimer' => 0.02, 'refuse' => 0.01,
], confidence: 0.95),
]]);
$verdict = app(AnswerVerifier::class)->verify(
'When does the trial end?',
'The trial ends after 14 days.',
collect(['Trials run for 14 days.']),
);
expect($verdict->action())->toBe($expected);
})->with([
'clean' => [0.97, Action::Pass],
'drifted' => [0.70, Action::Regenerate],
'invented' => [0.10, Action::Refuse],
]);
it('sends the passages to the verifier', function () {
Classification::fake();
app(AnswerVerifier::class)->verify('q', 'a', collect(['Trials run for 14 days.']));
Classification::assertClassified(
fn (ClassificationPrompt $prompt) => $prompt->contains('14 days')
&& $prompt->asks('grounded'),
);
});
Pair this with DocsAnswerer::fake() when you are testing the middleware end to end — the patterns are in faking Laravel AI SDK agents in Pest.
Gotchas and Edge Cases#
Classification is experimental. The SDK docs flag it as such, and the API may change in a minor release. Keep the calls behind AnswerVerifier so a signature change is one file, not forty.
confidence can be null. A Choice answer's confidence is null when the provider cannot measure it. $this->policyConfidence ?? 0.0 in the verdict is not defensive padding — a null there with a naive comparison is a TypeError in production at 3am.
Send retrieved passages, not the corpus. The state is billed as input tokens, and a bigger haystack makes the groundedness question harder, not easier. Six passages verify better than sixty.
Boolean answers carry no confidence. Only Choice and Score do. A two-outcome distribution describes itself, so you read uncertainty as distance from 0.5 rather than looking for a separate field.
Watch the block rate, not just the blocks. A verifier that silently starts refusing 40% of answers after a retrieval change looks exactly like a quiet product. Alert on the rate. The same instinct applies to running LLM evals against a golden dataset — the guardrail tells you something broke, the eval suite tells you what.
Structured output is not verification. A schema guarantees the shape of the answer, never its truth. JSON schema structured output and this guardrail solve different problems and you want both.
Wrapping Up#
Install laravel/ai, set TYPESAFE_API_KEY, write three questions and a match expression. That is a grounding check on every answer for a fraction of a cent, in a fraction of a second, from a model that fails independently of the one that wrote the text.
Start with grounded alone at a high threshold, log every verdict for a week, then add questions and lower thresholds from what the logs show. From here, LLM evals with a golden dataset in CI catch regressions before deploy, and input guardrails and moderation close the other side of the loop.
FAQ#
How do I stop an LLM from hallucinating in Laravel?
You cannot stop it, so catch it instead. Generate the answer as normal, then ask a separate verifier whether every claim in that answer is supported by the source passages you retrieved, and refuse or regenerate when the probability is low. In the Laravel AI SDK this is a Classification call with a Boolean question, backed by TypeSafe's Jev model, and it runs in a fraction of a second on every response.
How do I validate AI-generated answers before showing them to users?
Verify the answer against the material it was supposed to use, not against the model's own confidence. Pass the question, the answer and the retrieved passages as structured state, ask a groundedness question and a citation-support question in the same request, and let your own code turn the returned probabilities into pass, regenerate or refuse. Keep the thresholds in PHP where you can read them, test them and change them without touching a model.
Can one AI model check another model's output?
Yes, and it works better when the checker is a different kind of model. A second LLM grading the first costs a full generation and tends to make correlated mistakes, because it is the same architecture reading the same text. A System One model like Jev returns a calibrated probability rather than prose, answers far faster, and prices input tokens at $0.042 per million with output free, which is what makes checking every single response affordable.
Does the Laravel AI SDK have built-in guardrails?
It has the pieces rather than a single guardrail feature. Agent middleware intercepts every generation step, Classification gives you typed Boolean, Choice and Score questions backed by TypeSafe, tool approvals put a human in front of destructive actions, and structured output constrains the response shape. Composing those into a policy — which checks run, what the thresholds are, what happens on a failure — is deliberately left to your application.