Does read-back actually catch errors? We measured.
The moat only matters if it works. So we ran the real pipeline on a controlled set with known answers — and published the numbers, the method, and the limits.
This is a small, controlled benchmark we ran ourselves — not a field study. The point is that the numbers are verifiable facts, and the method and its limits are laid out in full below.
What read-back is
After translating each string, KAERIS runs three checks — two deterministic, one model-based:
- Placeholder & tag integrity. A regex-level check that every
{name},%s,%(count)s,<b>,<0>and ICU plural in the source survived. This is a fact, not a judgment. - Meaning drift (the distortion judge). Each translation is independently translated back into your language by a different model, then a judge compares the original against that back-translation and flags genuine meaning changes — a flipped negation, a wrong number, an opposite action.
The forward translation and the back-translation use different models, so a model's own mistake doesn't get a free pass when it grades itself.
How we tested it
Test A — placeholder & tag integrity (end-to-end, real model)
20 realistic UI strings carrying a mix of placeholder styles (Rails %{name}, Python %(name)s, printf %.2f/%02d, .NET {0}, HTML <b>, i18next <0>, ICU plurals) were translated into four languages spanning different scripts — German, Russian, Japanese and Arabic — by the production free model (DeepSeek V3). That's 80 translations. We then ran the deterministic check on the real model output.
Result: the model dropped a placeholder in 3 of 80 translations (~4%). Read-back flagged all 3 — the check is deterministic, so every real loss is caught by construction. Modern models are good at this; read-back is the safety net for the ~4% that still slips.
Test B — the distortion judge (known ground truth)
We authored 28 back-translations that clearly change meaning (negation lost, wrong number, opposite action, required→optional, and similar) and 40 back-translations that keep the meaning but reword heavily (the hard case for false positives). The judge (Gemini 2.5 Flash, independent of the forward model) saw them with no labels.
- Recall: 28/28 (100%) — every planted meaning error was flagged, with an accurate reason ("negation lost", "wrong number", "opposite meaning").
- False-positive rate: 0/40 (0%) — not one correctly-reworded paraphrase was flagged.
What the benchmark also fixed
Running this surfaced a real bug in our own pipeline: on a large batch, the judge call could exceed its timeout and — being fail-silent — return no flags, quietly under-reporting on big files. We now split judge calls into small sub-batches with a retry, so verify stays reliable regardless of file size. That fix shipped with this page. Finding it is exactly what an honest benchmark is for.
Honest limits
We'd rather you trust the number because you can see its edges:
- This is a small, controlled set (148 cases), not a sample of live traffic.
- The 28 distortions are clear meaning changes. Subtle drift — a shifted tone, a softened modal — is harder, and 100% here does not mean 100% on everything.
- We wrote the test cases, so the ground truth is known. That's a fair test of the judge, not a field study of translation quality.
- One forward model (DeepSeek V3) and one judge model (Gemini 2.5 Flash); the distortion set is English-centric.
The claim we stand behind: read-back's flags are verifiable facts — a lost placeholder, a flipped negation — not a confidence score. Better translation models don't shrink that value; they just make the safety net quieter.
Free, no account. Turn on verify and read every line back in your own language.
Try the read-back →