Are These Two Customers the Same Person? Entity Resolution with Jev

Every CRM I have been handed eventually holds the same customer three times. The first fix is always similar_text() over the name column, and it fails in both directions at once — it misses "Robert" against "Bob" and it cheerfully merges two branches of the same firm. The judgment you actually want is "would someone who knows this data say these are the same entity?", which is a snap judgment, and that is exactly what a System One model answers with a number you can threshold.

See why fuzzy string matching fails in both directions#

Start by proving the baseline is broken, because someone will ask you to justify replacing it. Take four pairs out of a real table and run a string metric over the names. Two of them are the same entity and score badly; two of them are different entities and score well. No threshold separates them.

$pairs = [
    // Same company. Nickname plus legal suffix variation.
    ["Bob's Plumbing Ltd", 'Robert Plumbing Limited'],
    // Same person. First and last name transposed on import.
    ['Nguyen Thi Mai', 'Mai Thi Nguyen'],
    // DIFFERENT companies. Two branches of one franchise.
    ['Smith & Sons (Leeds)', 'Smith & Sons (York)'],
    // DIFFERENT people. Father and son at one address.
    ['James Okonkwo Jr', 'James Okonkwo'],
];

foreach ($pairs as [$a, $b]) {
    similar_text($a, $b, $percent);
    printf("%-24s %-24s %5.1f%%\n", $a, $b, $percent);
}

The two pairs you want to merge come back at roughly 61% and 71%. The two pairs that must never merge come back at 84% and 93%. The metric is not weakly wrong, it is inverted — it is measuring the wrong thing. Levenshtein distance has no concept of "Ltd and Limited are the same word" or "Leeds and York are different places", and a weighted rules engine only papers over that by accumulating a new clause every time somebody complains.

What distinguishes these pairs is everything except the name: postcode, email domain, phone digits, the date the account was created. So send the model the whole record, not the name.

Block candidate pairs in SQL#

Comparing every customer with every other customer is O(n²) — 120,000 customers is 7.2 billion pairs, and you are not paying for that. Blocking narrows it to pairs that share something cheap and indexable, and you only ever ask the model about what survives.

Normalise the blocking keys into stored columns first so the index is usable. A generated column or an observer both work; the point is that UPPER(REPLACE(postcode, ' ', '')) must not appear in the WHERE clause.

Schema::table('customers', function (Blueprint $table): void {
    $table->string('postcode_key', 10)->nullable()->index();
    $table->string('email_domain')->nullable()->index();
    $table->string('phone_key', 15)->nullable()->index();  // digits only, last 9
    $table->foreignId('merged_into_id')->nullable()->constrained('customers');
});

Then one self-join per blocking key, unioned, with a.id < b.id so each pair appears once and never pairs a row with itself. The tenant equality is in the join condition, not in the model's judgment — more on that below.

<?php

declare(strict_types=1);

namespace App\Dedupe;

use Illuminate\Support\Facades\DB;
use Illuminate\Support\Collection;

final class CandidatePairGenerator
{
    /**
     * @return Collection<int, object{left_id: int, right_id: int, block: string}>
     */
    public function forTenant(int $tenantId): Collection
    {
        $block = fn (string $column, string $label) => DB::table('customers as a')
            ->join('customers as b', function ($join) use ($column): void {
                $join->on("a.{$column}", '=', "b.{$column}")
                    ->on('a.id', '<', 'b.id')
                    // Tenancy is a hard SQL precondition, never a model judgment.
                    ->on('a.tenant_id', '=', 'b.tenant_id');
            })
            ->where('a.tenant_id', $tenantId)
            ->whereNull('a.merged_into_id')
            ->whereNull('b.merged_into_id')
            ->whereNotNull("a.{$column}")
            ->selectRaw('a.id as left_id, b.id as right_id, ? as block', [$label]);

        return $block('postcode_key', 'postcode')
            ->union($block('email_domain', 'email_domain'))
            ->union($block('phone_key', 'phone'))
            ->get();
    }
}

Measure the reduction and put the number in your commit message. On the table I built this against, 118,400 customers gave 7.0 billion theoretical pairs and 9,214 blocked pairs — a reduction of roughly six orders of magnitude, at 340ms of query time. That is the whole economic argument for blocking: the model call is the expensive part, so the SQL decides your bill.

Blocking on email_domain alone is noisy because every gmail.com customer in the tenant blocks against every other one. Either pair it with a second condition, or drop free-mail domains from that block and let the postcode and phone blocks carry those records.

Design the Jev questions#

Jev evaluates every question in a request against the same state, in parallel, and output tokens are free — so ask the judgment question and the questions that explain it. One Score for "are these the same entity", three Nouls naming the specific disagreements a reviewer needs to see.

The Score levels are ordered, and the model returns a probability-weighted value that can land between them. Write the levels as things a human would say, not as numbers.

<?php

declare(strict_types=1);

namespace App\Dedupe;

final class PairQuestions
{
    /**
     * @return array<string, array<string, mixed>>
     */
    public static function all(): array
    {
        return [
            'same_entity' => [
                'type' => 'score',
                'instructions' => 'Are record A and record B the same real-world customer?',
                'criteria' => [
                    'Clearly different entities — different people or different companies',
                    'Probably different, but some fields coincide',
                    'Genuinely ambiguous — could be either',
                    'Probably the same entity with data-entry variation',
                    'Certainly the same entity',
                ],
            ],
            'different_address' => [
                'type' => 'noul',
                'instructions' => 'Do the two records describe physically different addresses?',
                'criteria' => [
                    'true' => 'Different street, town or premises — not a formatting difference',
                    'false' => 'The same address written two ways, or one is missing',
                ],
            ],
            'different_contact' => [
                'type' => 'noul',
                'instructions' => 'Do the two records belong to different named individuals?',
                'criteria' => [
                    'true' => 'Two distinct people, including a parent and child sharing a name',
                    'false' => 'One person, or a nickname, initial or transposition of one name',
                ],
            ],
            'different_legal_entity' => [
                'type' => 'noul',
                'instructions' => 'Are these two separate legal entities rather than one company?',
                'criteria' => [
                    'true' => 'Separate branches, subsidiaries or company numbers',
                    'false' => 'One company written with different suffixes or trading names',
                ],
            ],
        ];
    }
}

A Score answer comes back as score plus confidence plus a legend mapping level indices to your text. A Noul comes back as a single noul float from 0 to 1 with no confidence attached — the two-outcome distribution already describes itself, so you test distance from the middle. That asymmetry matters in the next step.

The state is the full record on both sides. Sending only names and asking about identity produces confident nonsense, because you have removed every field that distinguishes.

$state = [
    'record_a' => $left->only(['name', 'contact_name', 'email', 'phone', 'address_line_1', 'postcode', 'company_number', 'created_at']),
    'record_b' => $right->only(['name', 'contact_name', 'email', 'phone', 'address_line_1', 'postcode', 'company_number', 'created_at']),
];

Turn confidence into merge, review or leave#

Two numbers gate the decision: the Score, which says how same they are, and the confidence, which says how sure the model is of that reading. Act automatically only when both are high. The shape here is the same one I use for confidence-gated support ticket routing, with the thresholds pulled much tighter because the action is destructive.

<?php

declare(strict_types=1);

namespace App\Dedupe;

enum PairVerdict: string
{
    case Merge = 'merge';
    case Review = 'review';
    case Distinct = 'distinct';

    /**
     * @param  array<string, array<string, mixed>>  $answers
     */
    public static function fromAnswers(array $answers): self
    {
        $score = (float) $answers['same_entity']['score'];
        $confidence = (float) $answers['same_entity']['confidence'];

        // Any hard disagreement vetoes an automatic merge outright.
        $vetoed = (float) $answers['different_contact']['noul'] > 0.70
            || (float) $answers['different_legal_entity']['noul'] > 0.70;

        return match (true) {
            // Level 4 is "certainly the same entity" and the model is sure of it.
            $score >= 3.6 && $confidence >= 0.92 && ! $vetoed => self::Merge,
            $score >= 2.0 => self::Review,
            default => self::Distinct,
        };
    }
}

The veto is doing real work. "James Okonkwo Jr" and "James Okonkwo" at one address will score high on same_entity because almost every field matches — the only thing separating them is that they are two people, which is precisely what different_contact asks. A high Score with a high different_contact is not a contradiction to resolve in the prompt, it is a signal to route the pair to a human.

Store the full answer payload against the pair whatever the verdict. The Distinct decisions are the ones that stop you re-asking the same question every night, and the whole set is your threshold-tuning dataset later.

Batch pairs into requests#

One pair per request pays for nothing but overhead. Jev bills per input token at $0.042 per million with output free, and on Jev 1.13 a request can total 64k tokens across the state and every question combined, with a further 32k ceiling on the state plus the longest single question. A pair of customer records is small, so you can fit a lot of them into one call by namespacing the questions per pair.

<?php

declare(strict_types=1);

namespace App\Dedupe;

use Illuminate\Support\Facades\Http;

final readonly class PairJudge
{
    public function __construct(
        private string $apiKey,
        private string $model = 'jev-latest',
    ) {}

    /**
     * @param  array<string, array{record_a: array<string, mixed>, record_b: array<string, mixed>}>  $pairs
     * @return array<string, array<string, array<string, mixed>>>  keyed by pair key
     */
    public function judge(array $pairs): array
    {
        $questions = [];

        foreach ($pairs as $key => $pair) {
            foreach (PairQuestions::all() as $name => $question) {
                // Point each question at its own slice of state.
                $question['instructions'] = "For pair {$key}: {$question['instructions']}";
                $questions["{$key}__{$name}"] = $question;
            }
        }

        $answers = Http::withToken($this->apiKey)
            ->timeout(30)
            // 429 and 529 are the documented back-off cases; 401 and 422 are not.
            ->retry(3, fn (int $attempt): int => $attempt * 250, throw: false)
            ->post('https://api.typesafe.ai/v1/systemone', [
                'state' => $pairs,
                'model' => $this->model,
                'questions' => $questions,
            ])
            ->throw()
            ->json('answers');

        return $this->regroup($answers);
    }

    /**
     * @param  array<string, array<string, mixed>>  $answers
     * @return array<string, array<string, array<string, mixed>>>
     */
    private function regroup(array $answers): array
    {
        $grouped = [];

        foreach ($answers as $compositeKey => $answer) {
            [$pairKey, $question] = explode('__', $compositeKey, 2);
            $grouped[$pairKey][$question] = $answer;
        }

        return $grouped;
    }
}

Size the batch by tokens, not by a round number of pairs. Twenty pairs of four questions is eighty questions in one request, which is fine on the question count — there is no documented cap — but a tenant with long address fields will hit the 64k token budget sooner than you expect. Serialise the state, measure it, and split when you cross about 70% of the budget. The economics of this are the same ones I walked through in batching Jev questions into one request: the state is what you pay for, so the question is how many times you send it.

Merge behind a reversible, audited action#

This is the dangerous part and deserves the most careful code in the feature. A merge rewrites foreign keys across your schema, and "undo the merge we ran on 40,000 rows last Tuesday" is a sentence you never want to hear without a plan attached.

Four rules: pick the survivor deterministically, repoint children inside a transaction, keep the loser as a tombstone rather than deleting it, and write an audit row that a human can read.

<?php

declare(strict_types=1);

namespace App\Dedupe;

use App\Models\Customer;
use App\Models\CustomerMerge;
use Illuminate\Support\Facades\DB;

final class MergeCustomers
{
    /** Child tables whose customer_id follows the survivor. */
    private const CHILD_TABLES = ['orders', 'invoices', 'addresses', 'notes', 'subscriptions'];

    /**
     * @param  array<string, array<string, mixed>>  $evidence
     */
    public function handle(Customer $left, Customer $right, array $evidence, string $actor): CustomerMerge
    {
        abort_unless($left->tenant_id === $right->tenant_id, 422, 'Refusing a cross-tenant merge.');

        // Deterministic survivor: oldest record wins, id breaks the tie.
        [$survivor, $loser] = $left->created_at <= $right->created_at
            ? [$left, $right]
            : [$right, $left];

        return DB::transaction(function () use ($survivor, $loser, $evidence, $actor): CustomerMerge {
            $moved = [];

            foreach (self::CHILD_TABLES as $table) {
                $moved[$table] = DB::table($table)
                    ->where('customer_id', $loser->id)
                    ->update(['customer_id' => $survivor->id]);
            }

            // Tombstone, not deletion: the id stays resolvable forever.
            $loser->forceFill([
                'merged_into_id' => $survivor->id,
                'deleted_at' => now(),
            ])->save();

            return CustomerMerge::create([
                'tenant_id' => $survivor->tenant_id,
                'survivor_id' => $survivor->id,
                'loser_id' => $loser->id,
                'moved_counts' => $moved,
                'evidence' => $evidence,     // the full Jev answer payload
                'actor' => $actor,           // 'auto:jev-1.13.0' or a user id
                'lawful_basis' => 'legitimate_interest:data_accuracy',
            ]);
        });
    }
}

The unmerge is the same operation read backwards off the audit row, which is only possible because you recorded the counts and never deleted the loser.

public function unmerge(CustomerMerge $merge): void
{
    DB::transaction(function () use ($merge): void {
        foreach ($merge->moved_counts as $table => $count) {
            DB::table($table)
                ->where('customer_id', $merge->survivor_id)
                // Only rows that were the loser's before the merge ran.
                ->where('updated_at', '>=', $merge->created_at)
                ->whereIn('id', $merge->movedIdsFor($table))
                ->update(['customer_id' => $merge->loser_id]);
        }

        $merge->loser()->withTrashed()->first()?->forceFill([
            'merged_into_id' => null,
            'deleted_at' => null,
        ])->save();

        $merge->delete();
    });
}

Record the moved ids, not just the counts, if your child tables are large — the count alone tells you how wrong the unmerge went, but not which rows to move back. Any job this merge fires should go out with ->afterCommit(), for the reasons in dispatching jobs after database transactions commit: a search-reindex job that runs before the merge commits will index the pre-merge state.

The middle band needs somewhere to go. A Filament page showing the two records side by side, with the three Noul answers rendered as plain sentences — "the model thinks these are different people (0.81)" — gives the reviewer the reason rather than a bare score. Building a custom Filament v5 page that hosts Livewire widgets covers the scaffolding for that screen.

Run it over a large table#

Blocked pairs still number in the thousands, so this is a queued batch, not an Artisan command you watch. Chunk by id, group pairs into batched requests, and make the write idempotent so a replayed job cannot merge twice.

<?php

declare(strict_types=1);

namespace App\Jobs;

use App\Dedupe\{MergeCustomers, PairJudge, PairVerdict};
use App\Models\{Customer, CandidatePair};
use Illuminate\Bus\Batchable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;

final class JudgeCandidatePairs implements ShouldQueue
{
    use Batchable, Queueable;

    public int $tries = 3;

    /** @param array<int, int> $pairIds */
    public function __construct(public array $pairIds) {}

    /** @return array<int, int> */
    public function backoff(): array
    {
        return [10, 60, 300];
    }

    public function handle(PairJudge $judge, MergeCustomers $merge): void
    {
        $pairs = CandidatePair::whereIn('id', $this->pairIds)
            // Idempotency: a replay skips anything already decided.
            ->whereNull('verdict')
            ->with(['left', 'right'])
            ->get();

        if ($pairs->isEmpty()) {
            return;
        }

        $answers = $judge->judge($pairs->mapWithKeys(fn (CandidatePair $pair): array => [
            "p{$pair->id}" => [
                'record_a' => $pair->left->toComparisonArray(),
                'record_b' => $pair->right->toComparisonArray(),
            ],
        ])->all());

        foreach ($pairs as $pair) {
            $answer = $answers["p{$pair->id}"];
            $verdict = PairVerdict::fromAnswers($answer);

            $pair->update(['verdict' => $verdict, 'evidence' => $answer]);

            if ($verdict === PairVerdict::Merge) {
                $merge->handle($pair->left, $pair->right, $answer, 'auto:jev-1.13.0');
            }
        }
    }
}

Dispatch it as a batch over chunks so a single bad pair does not sink the run, and so you get progress for free:

Bus::batch(
    CandidatePair::whereNull('verdict')
        ->pluck('id')
        ->chunk(20)                 // ~20 pairs per request
        ->map(fn ($ids) => new JudgeCandidatePairs($ids->all()))
        ->all()
)->allowFailures()->name('dedupe:'.$tenant->id)->dispatch();

Rate limiting matters once the batch is wide. Jev 1.13 publishes 80 requests per second and 100K tokens per second, and TypeSafe warns those are moving while they scale. A 429 carries a Retry-After header, so honour it rather than guessing — and if several workers are hitting the API at once, a job-level throttle is a cleaner home for that than a sleep(). The trade-offs between a batch and a chain here are the ones in Laravel queue chains vs batches.

Order the merges within a run. If A matches B and B matches C, merging A into B and then B into C leaves merged_into_id pointing at a tombstone. Resolve the chain before writing — follow merged_into_id until it is null, and merge into that record.

Measure precision, not accuracy#

Hand-label a few hundred pairs once. Pull a stratified sample across the score range, have someone who knows the data mark each pair same or different, and freeze that set as a fixture. Precision on merges is what matters: recall costs you a duplicate that stays in the table, precision costs you a destroyed customer record.

it('never merges a pair the reviewers called distinct', function (): void {
    $labelled = collect(json_decode(file_get_contents(base_path('tests/fixtures/dedupe-labels.json')), true));

    $falseMerges = $labelled
        ->filter(fn (array $row): bool => $row['label'] === 'distinct')
        ->filter(fn (array $row): bool => PairVerdict::fromAnswers($row['answers']) === PairVerdict::Merge);

    expect($falseMerges)->toBeEmpty();
});

it('auto-merges at least 60% of the pairs reviewers called identical', function (): void {
    $same = collect(/* ... */)->filter(fn (array $row): bool => $row['label'] === 'same');

    $auto = $same->filter(fn (array $row): bool => PairVerdict::fromAnswers($row['answers']) === PairVerdict::Merge);

    expect($auto->count() / $same->count())->toBeGreaterThan(0.60);
});

Because the fixture stores the raw Jev answers, those tests run without an API key and without a network call — they are testing your thresholds, which is the thing that actually changes. Run the live model against the labelled set separately on a schedule, the same way I run LLM evals as a golden dataset in Pest and CI, so you catch the day jev-latest moves underneath you.

Gotchas and Edge Cases#

Never let the model decide tenancy. Cross-tenant merging is a data breach, not a bug. Put a.tenant_id = b.tenant_id in the join and assert it again in the merge action, as above. If you are not sure how firmly your tenancy boundary holds under raw query-builder joins, Laravel 13 multi-tenancy with teams is worth a read before you point this at production.

GDPR follows the merge. A merge changes whose data sits where, and under what lawful basis. Record the basis on the audit row in words a DPO can read, keep the tombstone so a subject access request can still resolve the old id, and make sure an erasure request deletes both sides rather than only the survivor.

jev-latest is an alias. Thresholds tuned against one release are not automatically right for the next. Store the resolved version against every decision and pin the model explicitly in the job once your bands are calibrated.

Watch the error codes. A malformed or oversized question map comes back as 422 with the field named, not 413. 401 is a bad key. Only 429 and 529 are worth retrying, which is why the client above retries those rather than everything.

Free-mail domains are not a blocking key. gmail.com blocks every consumer customer against every other one and will quietly generate a million pairs. Exclude the top free-mail domains from that block or require a second match alongside it.

Wrapping Up#

Build it in this order: blocking first and measure the reduction, then the question set, then the labelled fixture, and only then let anything merge automatically. Start with the merge band set impossibly high so every pair lands in the review queue — a fortnight of human decisions tells you where the real threshold is far better than a guess does.

If you want the judgment itself to be more than one number, composite scoring with Jev shows how to combine several signals into a weighted decision, and batching Jev questions into one request is where the cost of all this is won or lost.

FAQ#

How do I find duplicate records in a Laravel database?

Do it in two passes. First generate candidate pairs in SQL with a self-join on normalised, indexed columns — postcode, email domain, the last nine digits of a phone number — using a.id < b.id so each pair appears once. That turns an O(n²) comparison into a few thousand pairs. Then judge only those pairs, either by hand or with a model, because the expensive step should only ever see records that already have something in common.

What is entity resolution?

Entity resolution is deciding whether two or more records refer to the same real-world thing — the same person, company, product or place — when there is no shared identifier to join on. It is distinct from string matching because the answer depends on the whole record rather than on one field: two identical names at different postcodes are usually different entities, and two very different names at one postcode with one phone number are usually the same one.

Is fuzzy string matching enough to deduplicate customers?

No, and it tends to be wrong in both directions at once. String distance misses "Bob" against "Robert" and "Ltd" against "Limited", while scoring two branches of the same franchise, or a father and son at one address, as near-identical. It has no model of what a name means. Use string similarity as one blocking signal among several, never as the decision.

How do I safely merge duplicate Eloquent models?

Wrap the whole thing in a database transaction, pick the surviving record by a deterministic rule such as oldest-created-wins, repoint every child table's foreign key to the survivor, and soft-delete the loser with a merged_into_id pointing at the survivor rather than hard-deleting it. Write an audit row recording which rows moved, who or what triggered the merge, and the evidence behind it. That row is what makes an unmerge possible.

How do I use Jev confidence to decide when a human should review a merge?

Gate on the Score and the confidence together, and automate only when both are high — a score at the top level with confidence above roughly 0.92 is a reasonable starting point for something destructive. Everything in the middle band goes to a review queue that shows the reviewer the companion Noul answers as the explanation, and everything below is recorded as distinct so it is not re-asked. Start the automatic band impossibly high and lower it once you have real review decisions to calibrate against.

How do I run deduplication over a large table without timing out?

Push it onto the queue as a batch rather than running it in a request or a long Artisan command. Chunk the candidate pairs, group roughly twenty pairs into each API request to stay inside the token budget, and dispatch one job per chunk with Bus::batch() and allowFailures(). Make the write idempotent by skipping pairs that already have a verdict, honour the Retry-After header on a 429, and resolve merge chains so a pair never merges into a record that is itself a tombstone.