Cut LLM Costs with a Semantic Response Cache in the Laravel AI SDK

A support bot I shipped last year was answering the same twelve questions all day, and paying full price every time. Cache::remember() did nothing, because "how do I cancel my subscription" and "can I cancel my plan?" are different cache keys but the same question. The fix was caching by meaning instead of by string.

Why exact-match caching fails for LLM prompts#

Traditional caching hashes the input. That works when the input is a user ID or a query string, because the same request produces byte-identical keys. Natural language does not behave that way.

Take three prompts that want the same answer:

How do I cancel my subscription?
Can I cancel my plan?
what's the process for cancelling

Three different MD5 hashes. Three cache misses. Three completions billed. On a support bot handling a few thousand messages a day, that is real money — and the token usage tracking will show you exactly how much, which is usually the moment someone asks whether any of it can be avoided.

You could normalise the string — lowercase it, strip punctuation, sort the words. I've tried. It buys you a handful of extra hits and breaks the moment someone writes a full sentence.

How a Laravel AI semantic cache works#

The idea is short enough to describe in four steps:

1. Embed the incoming prompt into a vector.
2. Search stored prompt vectors for the nearest neighbour.
3. If the best match is above a similarity threshold, return its stored completion.
4. Otherwise call the model, then store the prompt, its embedding, and the answer.

The only genuinely new part is step 2, and Laravel 13 does it for you. This is not the same thing as a RAG pipeline with pgvector — RAG embeds your documents to build context for a prompt. A semantic cache embeds the prompt to skip the model call entirely. Same infrastructure, opposite direction.

Store prompt embeddings in pgvector#

Vector queries in Laravel need PostgreSQL with the pgvector extension (or MongoDB via the Laravel MongoDB package). If you're on Laravel Cloud, pgvector is already installed on every Postgres database; otherwise Schema::ensureVectorExtensionExists() will enable it for you. I've written up the serverless Postgres and pgvector setup separately if you're starting from nothing.

The migration:

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::ensureVectorExtensionExists();

        Schema::create('prompt_cache_entries', function (Blueprint $table) {
            $table->id();
            $table->foreignId('team_id')->nullable()->index();
            $table->string('agent')->index();
            $table->text('prompt');
            $table->longText('completion');
            // Must match the dimension count of the embedding model you use below.
            $table->vector('embedding', dimensions: 1536)->index();
            $table->timestamp('expires_at')->index();
            $table->timestamps();
        });
    }

    public function down(): void
    {
        Schema::dropIfExists('prompt_cache_entries');
    }
};

Calling index() on a vector column creates an HNSW index with cosine distance. You want it — without it every lookup is a sequential scan over every row you have ever cached.

The model casts the vector column to an array so Laravel converts between PHP arrays and Postgres vector values for you:

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Model;

class PromptCacheEntry extends Model
{
    protected $guarded = [];

    protected function casts(): array
    {
        return [
            'embedding' => 'array',
            'expires_at' => 'datetime',
        ];
    }
}

Pull the tuning knobs into config rather than scattering magic numbers through the service:

<?php

// config/prompt-cache.php

return [
    'enabled' => env('PROMPT_CACHE_ENABLED', true),
    'min_similarity' => (float) env('PROMPT_CACHE_MIN_SIMILARITY', 0.95),
    'ttl_hours' => (int) env('PROMPT_CACHE_TTL_HOURS', 24),
    'dimensions' => 1536,
    'embedding_model' => 'text-embedding-3-small',
];

Look up a cached answer by similarity#

whereVectorSimilarTo compares a query embedding against a stored vector column using cosine similarity, filters out anything below minSimilarity, and orders what's left most-similar-first. That last part matters: it means ->first() gives you the best match, not an arbitrary one.

$hit = PromptCacheEntry::query()
    ->where('agent', $agent)
    ->where('expires_at', '>', now())
    ->whereVectorSimilarTo('embedding', $embedding, minSimilarity: 0.95)
    ->first();

Similarity runs from 0.0 to 1.0, where 1.0 means the vectors are identical. Standard where clauses compose with it normally, which is how the per-agent and per-tenant scoping below works.

You can pass a plain string instead of an embedding array and Laravel will generate the embedding for you. I don't, in a cache — I want the embedding in hand so I can store it on a miss without paying for a second embedding call.

Wrap the Laravel AI SDK call with the semantic cache#

Here's the whole thing. It takes a closure so the caller decides how the completion is produced — a dedicated agent class, an anonymous agent, whatever.

<?php

namespace App\Ai\Cache;

use App\Models\PromptCacheEntry;
use Closure;
use Laravel\Ai\Embeddings;
use Laravel\Ai\Enums\Lab;

class SemanticPromptCache
{
    /**
     * Resolve a completion, reusing a stored answer when a semantically
     * similar prompt has already been answered for this agent and tenant.
     *
     * @param  Closure(): string  $resolver
     */
    public function remember(
        string $agent,
        string $prompt,
        Closure $resolver,
        ?int $teamId = null,
    ): string {
        if (! config('prompt-cache.enabled')) {
            return $resolver();
        }

        $embedding = $this->embed($prompt);

        $hit = PromptCacheEntry::query()
            ->where('agent', $agent)
            // Never leak one tenant's answer into another tenant's request.
            ->when($teamId, fn ($query) => $query->where('team_id', $teamId))
            ->where('expires_at', '>', now())
            ->whereVectorSimilarTo(
                'embedding',
                $embedding,
                minSimilarity: config('prompt-cache.min_similarity'),
            )
            ->first();

        if ($hit) {
            return $hit->completion;
        }

        $completion = $resolver();

        PromptCacheEntry::create([
            'team_id' => $teamId,
            'agent' => $agent,
            'prompt' => $prompt,
            'completion' => $completion,
            'embedding' => $embedding,
            'expires_at' => now()->addHours(config('prompt-cache.ttl_hours')),
        ]);

        return $completion;
    }

    /**
     * Generate the embedding vector for a prompt.
     *
     * @return list<float>
     */
    protected function embed(string $prompt): array
    {
        return Embeddings::for([$prompt])
            ->dimensions(config('prompt-cache.dimensions'))
            // Byte-identical prompts skip the embedding API call entirely.
            ->cache(seconds: 3600)
            ->generate(Lab::OpenAI, config('prompt-cache.embedding_model'))
            ->embeddings[0];
    }
}

Calling it from a controller:

$question = $request->string('question')->toString();

$answer = app(SemanticPromptCache::class)->remember(
    agent: SupportBot::class,
    prompt: $question,
    resolver: fn () => (string) SupportBot::make(user: $request->user())
        ->prompt($question),
    teamId: $request->user()->current_team_id,
);

Note the ->cache(seconds: 3600) on the embeddings call. That's the AI SDK's own embedding cache, keyed on provider, model, dimensions and input. It handles the byte-identical case for free, so a repeated prompt costs you nothing at all — not even the embedding.

The obvious question is whether this belongs in agent middleware instead. It can, and the ergonomics are nicer. I keep it as an explicit service because a cache that silently short-circuits an agent is a cache that eventually confuses someone at 2am, and because the closure form composes cleanly with queued background agent runs.

Testing it without touching a provider

Embeddings::fake() generates vectors of the right dimensions for every prompt, which is enough to exercise the miss path. For the hit path, hand it the same vector twice:

use Laravel\Ai\Embeddings;

it('serves a cached completion for a semantically similar prompt', function () {
    $vector = Embeddings::fakeEmbedding(1536);

    Embeddings::fake([[$vector], [$vector]]);

    $cache = app(SemanticPromptCache::class);
    $calls = 0;

    $resolver = function () use (&$calls) {
        $calls++;

        return 'Open Settings, then Billing, then Cancel.';
    };

    $cache->remember('SupportBot', 'How do I cancel my subscription?', $resolver);
    $cache->remember('SupportBot', 'Can I cancel my plan?', $resolver);

    expect($calls)->toBe(1);
});

The wider pattern of faking agents in Pest applies here too — assert on what the cache did, not on what the model said.

Choosing a threshold, TTL, and invalidation#

The threshold is the only number that can hurt you. Too high and you never hit; too low and you confidently return the answer to a different question.

I start at 0.95 and tune down. In practice:

• 0.98+ — effectively paraphrase-only. Safe, low hit rate.
• 0.95 — catches genuine rewordings. My default for support and FAQ traffic.
• 0.90 — noticeably higher hit rate, and the point at which I want a human to review a sample of hits before it goes anywhere near production.
• Below 0.90 — I haven't found a case where this was worth it. "How do I cancel?" and "How do I upgrade?" are closer in vector space than you'd like.

Log the similarity score of every hit for the first few weeks. selectVectorDistance will give you the raw distance as a column if you want to record it. You will find your threshold faster from a histogram of real traffic than from anything I can tell you.

For TTL, the question isn't "how long is an answer good for" — it's "how fast does the data behind the answer change". Pricing questions might be safe for a week; anything touching a live account balance shouldn't be cached at all. The expires_at column keeps expired rows out of lookups, and Laravel's prunable models will delete them on a schedule so the table doesn't grow without limit.

Event-driven invalidation is worth the extra work when your source of truth is a document set: when a knowledge-base article is updated, delete cache entries whose stored prompts are semantically near that article. Same query, run in the other direction.

Gotchas and Edge Cases#

Dimension mismatches fail at the database, not in PHP. The vector column is declared with a fixed dimension count. Switch embedding models — or change the dimensions() argument — and inserts start failing against existing rows. Changing model means a new column or a truncated table, not a config tweak.

Vector search is Postgres-only. whereVectorSimilarTo requires PostgreSQL with pgvector, or MongoDB with the Laravel MongoDB package. There is no MySQL path. If your app is on MySQL, this article is a case for a second datastore, not for a workaround.

Personalised agents are not cacheable across users. If your agent's instructions interpolate the user's name, plan, or account state, the prompt is only half the input. Cache the answer and you'll hand one user another user's context. Scope by tenant at minimum, and skip the cache entirely for agents whose system prompt varies per user.

Conversational agents need the history in the key. The same follow-up question means something different depending on what came before it. If you're using conversation memory, embed the last turn or two along with the prompt, or don't cache multi-turn agents at all.

Streaming responses complicate the hit path. A cached hit returns instantly, which is great, but if your frontend expects a token stream you'll need to replay the cached string as one. Not hard, just easy to forget until the UI breaks.

Non-deterministic prompts poison the cache. "What's the weather today" cached for 24 hours is a bug. Anything time-sensitive, personalised, or tool-driven should bypass the cache — an allowlist of cacheable agents beats a denylist of prompts.

Wrapping Up#

Add the migration, the model, and the service, set the threshold to 0.95, and log every hit for a fortnight before you touch the number. On repetitive support traffic I've seen this take 30–40% of completions off the bill, and the hits come back in milliseconds instead of seconds.

Pair it with token usage and cost tracking so you can prove the saving, and read the complete guide to the Laravel AI SDK if you're still deciding how the rest of your AI layer should be structured.

FAQ#

What is a semantic cache for LLM calls?

A semantic cache stores previous prompts and their completions alongside a vector embedding of each prompt. When a new prompt arrives, it's embedded and compared against the stored vectors. If a past prompt is close enough in meaning, the stored answer is returned without calling the model. It trades a cheap embedding call for an expensive completion call.

How is semantic caching different from normal caching?

Normal caching keys on the exact input — hash the string, look it up, hit or miss. Semantic caching keys on meaning, so "can I cancel my plan?" can hit an entry stored for "how do I cancel my subscription?". The trade-off is that a normal cache is exact and a semantic cache is probabilistic: you choose a similarity threshold, and the wrong threshold returns the right answer to the wrong question.

How do I cache AI SDK responses in Laravel?

Generate an embedding for the prompt with Embeddings::for([$prompt])->generate(), store it in a vector column on Postgres with pgvector, and query it with whereVectorSimilarTo('embedding', $embedding, minSimilarity: 0.95). On a hit, return the stored completion; on a miss, prompt the agent and insert a new row with the prompt, embedding, answer and an expiry timestamp. No extra package is required beyond laravel/ai.

What similarity threshold should I use for a prompt cache?

Start at 0.95 and adjust from real traffic. Above 0.98 you'll only catch near-identical rewordings and the hit rate stays low. Around 0.90 the hit rate climbs but semantically adjacent questions — cancel versus upgrade, for example — start colliding. Log the similarity of every hit for a few weeks and pick the threshold from the distribution rather than from a guess.

When should I not use a semantic cache?

Skip it for anything time-sensitive, personalised, or tool-driven: live account data, prompts that interpolate user state, agents whose answers depend on the current date, and multi-turn conversations where the same question means different things in different contexts. Also skip it when your traffic is genuinely diverse — if no two prompts are ever similar, you're paying for embeddings and getting nothing back.