A knowledge base import landed on me last month: eleven thousand articles exported from a legacy CMS as plain text, hard-wrapped at 72 columns, with every heading, bullet and warning box flattened into bare lines. The obvious move is to convert plain text to Markdown in Laravel by handing each document to a text model and asking for formatting. I tried it on fifty documents and diffed the output. Nine of them came back with a sentence quietly reworded, and one had a product name "corrected" to a competitor's.
Diff the LLM output before you convert plain text to Markdown#
Here is the shape of the problem, from the import. The source line reads:
Customers on the Pro tier may request a refund within 14 days of the
renewal date, provided the account has not exceeded 500 API calls in
that period.
The generated "formatted" version:
Customers on the Pro plan can request a refund within 14 days of their renewal date, as long as the account hasn't exceeded 500 API calls during that period.
Tier became plan. May became can. Provided became as long as. The meaning survived; the contract did not. For anything legal, medical, financial or contractual, that output is unshippable, and no amount of "do not change the wording" in the prompt makes it reliably safe — you are asking a text generator not to generate text.
The rule for the rest of this article is the one I used for extraction that cannot hallucinate: the model only ever answers questions about the text, and your code does every byte of the writing. Jev is a System One model — it returns judgments, never prose — so it cannot paraphrase a clause even if you ask it to. The structure comes from the model. The characters come from your input.
Two questions get you from plain text to Markdown. Did this line break split a sentence? And what kind of content is this block? That is the whole pipeline.
Read the markers your code can already see#
Before a single request goes out, read the evidence that is still in the file. Blank lines survive almost every flattening. So do stray - , 1. , # and > markers, because most exports only strip styling, not every character. Indentation survives too.
Every one of those is a decision your code can make for free, and a question you then do not have to ask. That matters twice over: it cuts the request size, and it removes a class of wrong answers, because a model asked to reconsider a blank line will occasionally get it wrong.
<?php
declare(strict_types=1);
namespace App\Support\Markdown;
final readonly class PairEvidence
{
private const MARKER = '/^\s*(?:[-*+]\s|\d+[.)]\s|#{1,6}\s|>\s)/u';
/**
* Decide in PHP what PHP can see.
*
* false = this line starts a new block. null = genuinely ambiguous, ask Jev.
*/
public static function decide(string $previous, string $current): ?bool
{
if (trim($previous) === '' || trim($current) === '') {
return false; // A blank line on either side always ends the block.
}
if (preg_match(self::MARKER, $current) === 1) {
return false; // A surviving list, heading or quote marker starts a block.
}
if (str_starts_with($current, ' ') || str_starts_with($current, "\t")) {
return false; // Indented: code or a nested item, not a hard wrap.
}
return null; // The only case worth spending a question on.
}
}
On the knowledge base import this knocked out about 40% of the line pairs before pass one ran.
Stitch hard-wrapped lines with one Noul per pair#
Pass one asks a single yes/no question per ambiguous adjacent pair: does this line carry on a sentence that began on the one above it? Every pair goes in one request, because Jev ingests the state once and the questions are a rounding error on top of it — the economics I measured in batching thirteen Jev questions into one request. Sending one request per pair would be forty times the cost for identical answers.
Put the numbered lines in the state once, and have each question reference line numbers rather than repeating the text.
<?php
declare(strict_types=1);
namespace App\Support\Markdown;
use App\Support\Jev\Jev;
use App\Support\Jev\Noul;
use App\Support\Jev\QuestionSet;
final readonly class LineStitcher
{
public function __construct(private Jev $jev, private float $threshold = 0.6) {}
/**
* @param list<string> $lines
* @return array<int, bool> line index => true when it continues the previous line
*/
public function stitch(array $lines): array
{
$continuations = [];
$questions = QuestionSet::make();
foreach ($lines as $i => $line) {
if ($i === 0) {
continue;
}
$decided = PairEvidence::decide($lines[$i - 1], $line);
if ($decided !== null) {
$continuations[$i] = $decided;
continue;
}
$questions = $questions->add("pair_{$i}", new Noul(
instructions: "Does line {$i} of `document.lines` carry on a sentence that began on line "
. ($i - 1) . ', so that the break between them is a hard wrap rather than a new block?',
));
}
if (count($questions) === 0) {
return $continuations;
}
$answers = $this->jev->ask(['document' => ['lines' => $lines]], $questions);
foreach ($questions->ids() as $id) {
$index = (int) str_replace('pair_', '', $id);
// A Noul answer is a bare probability — there is no separate confidence field.
$continuations[$index] = $answers->noul($id) > $this->threshold;
}
return $continuations;
}
}
That Jev, QuestionSet, Noul and Answers set is the thin client from the batching article; ids() is a one-line addition to QuestionSet returning array_keys($this->questions). There is still no official PHP SDK for the System One API, so the Laravel HTTP client is the integration point.
Merging is ordinary PHP. The character a hard wrap replaced is a single space, so that is the character you put back:
<?php
declare(strict_types=1);
namespace App\Support\Markdown;
final readonly class BlockMerger
{
/**
* @param list<string> $lines
* @param array<int, bool> $continuations
* @return list<string>
*/
public static function merge(array $lines, array $continuations): array
{
$blocks = [];
$current = '';
foreach ($lines as $i => $line) {
if (trim($line) === '') {
if ($current !== '') {
$blocks[] = $current;
$current = '';
}
continue;
}
if ($current === '') {
$current = rtrim($line);
continue;
}
if ($continuations[$i] ?? false) {
$current .= ' ' . trim($line);
continue;
}
$blocks[] = $current;
$current = rtrim($line);
}
if ($current !== '') {
$blocks[] = $current;
}
return $blocks;
}
}
Classify each block with a Choice and its companion questions#
Pass two has to be a second request, and this is the part people try to collapse. The blocks you are classifying do not exist until pass one has answered — you cannot ask "what kind of content is block 4?" when block 4 is still three loose lines. Two requests is the floor, not a design choice you can optimise away.
Within pass two, though, fan out hard. Alongside the Choice that picks the block type, ask the companion questions that only some types need: how high a heading sits, whether a list item is one of a numbered sequence, which flavour of callout it is. They cost a few tokens each and they save a second round trip when the type comes back heading. This is the same speculative fan-out reasoning behind pairing a Choice with a companion Noul so the classifier can say none of the above.
<?php
declare(strict_types=1);
namespace App\Support\Markdown;
use App\Support\Jev\Answers;
use App\Support\Jev\Choice;
use App\Support\Jev\Jev;
use App\Support\Jev\Noul;
use App\Support\Jev\QuestionSet;
use App\Support\Jev\Score;
final readonly class BlockClassifier
{
public function __construct(private Jev $jev) {}
/** @param list<string> $blocks */
public function classify(array $blocks): Answers
{
$questions = QuestionSet::make();
foreach ($blocks as $i => $block) {
$questions = $questions
->add("type_{$i}", new Choice(
instructions: "What kind of content is block {$i} of `document.blocks`?",
criteria: [
'heading' => 'A short title or section label introducing what follows it.',
'paragraph' => 'Ordinary prose.',
'list_item' => 'One item belonging to a list of items.',
'quote' => 'Text quoted from another document, person or source.',
'code' => 'A command, path, snippet or configuration meant to be copied literally.',
'callout' => 'An aside the reader is meant to notice: a note, tip or warning.',
],
))
// Companion: read only when type is heading. Lowest level first.
->add("level_{$i}", new Score(
instructions: "How high in the document hierarchy does block {$i} of `document.blocks` sit, if it is a heading?",
criteria: [
'A minor sub-heading nested under two larger headings.',
'A sub-section heading under a larger section.',
'A top-level section heading.',
],
))
// Companion: read only when type is list_item.
->add("ordered_{$i}", new Noul(
instructions: "Is block {$i} of `document.blocks` one step in a sequence where the order matters, rather than an unordered bullet?",
))
// Companion: read only when type is callout.
->add("tone_{$i}", new Choice(
instructions: "If block {$i} of `document.blocks` is an aside, what is it warning or telling the reader?",
criteria: [
'note' => 'Neutral extra information.',
'tip' => 'A suggestion that makes something easier.',
'warning' => 'A consequence to avoid: data loss, cost, breakage.',
],
));
}
return $this->jev->ask(['document' => ['blocks' => $blocks]], $questions);
}
}
Four questions per block sounds extravagant until you price it. The state — the whole document — is sent once either way, and three extra judgments per block are tokens, not requests.
Render the Markdown in code, not in the model#
The renderer is the dullest class in the pipeline and that is the point. It maps a block plus its answers onto Markdown syntax and never touches the words.
<?php
declare(strict_types=1);
namespace App\Support\Markdown;
use App\Support\Jev\Answers;
final readonly class MarkdownRenderer
{
public function __construct(private float $minimumConfidence = 0.55) {}
/** @param list<string> $blocks */
public function render(array $blocks, Answers $answers): string
{
$rendered = [];
foreach ($blocks as $i => $block) {
$rendered[] = $this->renderBlock($i, $block, $answers);
}
return implode("\n\n", $rendered);
}
private function renderBlock(int $i, string $block, Answers $answers): string
{
// An unconfident Choice becomes a plain paragraph. Guessing is how you ship a wrong heading.
if ($answers->confidence("type_{$i}") < $this->minimumConfidence) {
return $this->escape($block);
}
return match ($answers->choice("type_{$i}")) {
// The article title owns h1, so headings start at two.
'heading' => str_repeat('#', $this->level($answers->score("level_{$i}"))) . ' ' . $block,
'list_item' => ($answers->noul("ordered_{$i}") > 0.5 ? '1. ' : '- ') . $this->escape($block),
'quote' => '> ' . $this->escape($block),
// Code is never escaped and never reflowed.
'code' => "```text\n{$block}\n```",
'callout' => '> [!' . strtoupper($answers->choice("tone_{$i}")) . "]\n> " . $this->escape($block),
default => $this->escape($block),
};
}
private function level(float $score): int
{
return match ((int) round($score)) {
2 => 2,
1 => 3,
default => 4,
};
}
/** Escape only what would change meaning at the start of a block. */
private function escape(string $text): string
{
return preg_replace('/^([#>*+-]|\d+[.)])(\s)/u', '\\\\$1$2', $text) ?? $text;
}
}
Escaping is where well-meaning renderers break fidelity. Escape every special character and a sentence containing 2 * 3 comes back with backslashes in it. Escape only the leading character that would otherwise become block syntax, and the body text passes through untouched.
Prove fidelity with a strip-and-compare test#
This is the test that makes the feature defensible to whoever signs off the import. Strip the Markdown you just generated and assert it reproduces the source text exactly, modulo whitespace. If it does not, the pipeline moved a character, and you want to know that in CI rather than in a contract.
<?php
declare(strict_types=1);
namespace App\Support\Markdown;
final class TextFidelity
{
public static function normalise(string $text): string
{
return trim(preg_replace('/\s+/u', ' ', $text) ?? $text);
}
public static function strip(string $markdown): string
{
$plain = preg_replace(
[
'/^```.*$/m', // fence lines
'/^>\s?\[![A-Z]+\]\s*$/m', // callout markers
'/^#{1,6}\s+/m', // heading hashes
'/^>\s?/m', // quote markers
'/^(?:[-*+]|\d+[.)])\s+/m', // list markers
],
'',
$markdown,
) ?? $markdown;
// Undo only the escapes the renderer could have added.
$plain = preg_replace('/\\\\([#>*+\-\d])/u', '$1', $plain) ?? $plain;
return self::normalise($plain);
}
}
<?php
use App\Support\Markdown\TextFidelity;
it('reproduces the source text exactly', function (string $fixture) {
$source = file_get_contents(base_path("tests/Fixtures/markdown/{$fixture}.txt"));
$markdown = file_get_contents(base_path("tests/Fixtures/markdown/{$fixture}.md"));
expect(TextFidelity::strip($markdown))->toBe(TextFidelity::normalise($source));
})->with(['refund-policy', 'runbook', 'release-notes']);
Run this over a golden set of twenty real documents and it catches renderer regressions, escape bugs and over-eager stitching in one assertion.
Convert plain text to Markdown in a queued job, not a controller#
Two sequential API calls over a document of any size is not controller work. Pass two is blocked on pass one, so the job is genuinely serial — and that is fine, because nobody is waiting on an HTTP response. The same reasoning applies to any multi-step model pipeline; I went through the retry and failure handling in detail in running background agent runs on the queue.
Cache on the document hash so a retried job after a 429 does not pay for pass one twice.
<?php
declare(strict_types=1);
namespace App\Jobs;
use App\Models\Document;
use App\Support\Jev\Jev;
use App\Support\Markdown\BlockClassifier;
use App\Support\Markdown\BlockMerger;
use App\Support\Markdown\LineStitcher;
use App\Support\Markdown\MarkdownRenderer;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Cache;
final class ReconstructMarkdown implements ShouldQueue
{
use Queueable;
public int $tries = 3;
public int $timeout = 180;
public function __construct(public readonly Document $document) {}
public function handle(Jev $jev, MarkdownRenderer $renderer): void
{
// xxh128 is fast and non-cryptographic, which is all a cache key needs.
$key = 'markdown:' . hash('xxh128', $this->document->body);
$markdown = Cache::remember($key, now()->addDay(), function () use ($jev, $renderer): string {
$lines = preg_split('/\R/u', $this->document->body) ?: [];
$continuations = (new LineStitcher($jev))->stitch($lines);
$blocks = BlockMerger::merge($lines, $continuations);
$answers = (new BlockClassifier($jev))->classify($blocks);
return $renderer->render($blocks, $answers);
});
$this->document->update([
'markdown' => $markdown,
'reconstructed_at' => now(),
]);
}
}
Long documents need chunking. The request budget covers the state plus every question combined, and a 4,000-line document produces thousands of pairs, so split on blank lines into windows of a couple of hundred lines and run the pair of passes per window. Overlap by one line so a hard wrap spanning a window boundary still gets asked about, and discard the duplicate answer. Chunked state also keeps each question's reference frame small, which measurably helps the stitching answers.
Flag low-confidence blocks before you publish#
Unconfident blocks render as plain paragraphs, which is the safe default — a paragraph that should have been a heading is ugly, a heading that should have been a paragraph is wrong. But safe is not the same as finished, so record which blocks fell below the threshold and put them in front of a human.
<?php
$review = collect($blocks)
->filter(fn (string $block, int $i): bool => $answers->confidence("type_{$i}") < 0.55)
->map(fn (string $block, int $i): array => [
'index' => $i,
'excerpt' => str($block)->limit(120)->value(),
'guess' => $answers->choice("type_{$i}"),
'confidence' => round($answers->confidence("type_{$i}"), 2),
])
->values()
->all();
A Livewire page showing the original beside the reconstruction, with those indexes highlighted, is what turned this from a script into something I was willing to run across eleven thousand documents. Keep the thresholds in config/jev.php rather than inline — the same argument I made for weighted composite scoring with Jev, where tuning a decision meant editing config, not code.
Sanitise the Markdown before you render it as HTML#
The Markdown you just built contains text from an untrusted source, and Str::markdown() passes raw HTML straight through by default. A legacy export full of <script> fragments that were harmless as plain text becomes a stored XSS the moment you convert it.
<?php
use Illuminate\Support\Str;
$html = Str::markdown($document->markdown, [
'html_input' => 'escape', // keep the characters, render them inert
'allow_unsafe_links' => false, // blocks javascript: and data: destinations
]);
Use escape rather than strip here. Stripping removes characters that were in the source document, which quietly breaks the fidelity guarantee the whole pipeline exists to provide.
Gotchas and Edge Cases#
Bare commands look like hard wraps. A shell command on its own line, under 72 characters, with no terminating full stop, is exactly the shape of a wrapped sentence. The indentation check catches the indented ones; for the rest, the code Choice option in pass two fixes the classification, but pass one may already have glued it to the paragraph above. If your corpus is command-heavy, add a cheap pre-pass regex for lines starting $ , sudo, composer, php artisan.
List items wrap too. A three-line bullet is a marker line followed by two continuation lines, and PairEvidence correctly refuses to ask about the marker line while the continuations go to Jev. Make sure your merger attaches continuations to the marker line rather than starting a new block — the code above does, because the marker only decides the pair it starts.
Smart quotes and non-breaking spaces survive. Legacy exports are full of \u{00A0} and curly quotes. Do not normalise them away: that is a character change. Normalise them only inside TextFidelity::normalise() so both sides of the comparison agree, and leave the rendered output alone.
Headings that are really short paragraphs. A one-sentence paragraph with no full stop reads as a heading to any classifier. The confidence gate is what handles this, not a better prompt. Do not lower the threshold to make the review queue shorter.
Do not reach for this when markers survived. If the source still has its # and - intact, or there is a deterministic parser for the format — reStructuredText, Textile, a CMS JSON export — use that. This pipeline is for text where the structure is genuinely gone.
Wrapping Up#
Build the evidence pass first and measure how many pairs it eliminates before you write a single question; on a wrapped-at-72 corpus it is usually a third or more. Then wire the two passes, and put the strip-and-compare test in CI before you let the job touch real records. If you want the request-shaping details underneath this — the state budget, the typed question objects, the Http::fake() patterns — start with batching Jev questions into one request. And if the Markdown you produce is going to end up on a content-driven site, a Markdown and Sushi-backed Laravel blog is where mine lives.
FAQ#
How do I convert plain text back into Markdown programmatically?
Split the job into two decisions rather than one generation. First decide which line breaks were hard wraps so you can merge lines back into blocks, then decide what kind of content each block is, and have your own code emit the Markdown syntax. The judgments can come from a model, but the characters must come from your input, otherwise you are rewriting the document rather than formatting it.
Why not just ask an LLM to format the text?
Because a text generation model generates text, and formatting is indistinguishable from light editing as far as it is concerned. In my own fifty-document sample, roughly one in five came back with a reworded clause, a dropped qualifier or a "corrected" proper noun, and none of it was flagged. Instruction tuning reduces the rate but does not remove it, which is unacceptable for contractual, medical or legal content.
How do I detect hard-wrapped lines in PHP?
Deterministically, you can only detect the cases where evidence survived: a blank line on either side, a surviving list or heading marker, or leading indentation all mean a new block rather than a wrap. Everything else is genuinely ambiguous and needs a judgment. A regex that guesses from sentence-ending punctuation and leading capitals gets the common case right and fails on abbreviations, bare commands and headings.
What is a Noul question in TypeSafe's System One API?
A Noul is the yes/no question type. You supply instructions describing the condition, and the answer comes back as a single calibrated probability that the condition holds. Unlike Choice and Score answers it carries no separate confidence field — the probability itself is the signal, so you compare it against a threshold you set rather than reading confidence.
How many API requests does structure recovery need?
Two per document, or per chunk of a long document. Pass one asks every ambiguous line pair in a single request, and pass two asks every merged block in a second. They cannot be combined, because the blocks that pass two classifies do not exist until pass one has answered. Everything inside each pass batches, so the request count does not grow with document length until you hit the token budget and chunk.
How do I sanitise AI-generated Markdown in Laravel?
Pass the converter options to Str::markdown(): set html_input to escape and allow_unsafe_links to false. Escaping renders any raw HTML in the source as visible, inert text, which preserves the original characters; stripping would delete them and break the fidelity guarantee. If the rendered HTML is shown to other users, pair that with a Content Security Policy rather than relying on the converter alone.