Open Decision Models Beat the Hosted API on Bilingual Guardrails

A customer wants to write a governance rule in plain language ("is this request trying to extract other people's personal data?"), attach a probability threshold, and have every prompt and model reply evaluated against it, in English and in Brazilian Portuguese. That is the product requirement we set out to support. The open question was the engine: which decision model should evaluate those rules?

We measured five of them on a purpose-built 2,195-sample bilingual corpus: Cloudflare's clef (27B) and clef-flash (9B), LiquidAI's d1-2B, a GLiFormer-based open server, and TypeSafe's hosted jev API as the proprietary reference. All speak the same SystemOne API: a text state plus named yes/no questions, answered with calibrated probabilities and no generated tokens.

The headline result: the quantized open-weights 27B matched or exceeded the hosted proprietary reference on nearly every rule and language we tested, and led on Brazilian Portuguese (jev keeps one narrow edge: hostile-tone AUROC 0.999 versus 0.997–0.998). The full corpus, harness, raw per-sample CSVs, and reproduction instructions are being released in the public Chrysent-AI/ai-benchmarks repository (benchmark data, harness, and corpus to be released soon).

Why decision models instead of an LLM judge

An LLM-as-judge spends generated tokens on an answer that should have been a probability. Decision models return typed answers directly: noul questions get P(yes), choices get a full probability distribution, scores get an ordered scale. No output parsing, no schema errors, no reasoning-token cost, and a latency profile that does not grow with output length. For governance rules (bounded decisions evaluated on every request), that shape is exactly right. The interesting engineering question is which model and which serving engine hold up.

The corpus: built to break optimistic benchmarks

Small guardrail corpora produce flattering numbers. Our first 100 hand-written samples gave one model a perfect score; the expanded corpus removed that illusion.

The final corpus has 2,195 synthetic samples across four rules (personal-data extraction, prompt injection, hostile tone, and confidential-information requests):

• 1,046 English, 1,047 pt-BR, 102 code-switched EN/PT messages
• 800 benign samples, of which 456 are deliberate near-misses: users updating their own data, public-financial questions that sound internal, frustrated-but-civil complaints, third-party PII quoted for summarization, and defensive security framing
• 1,395 violations with an obvious → moderate → subtle difficulty ladder (polite wrappers, faux authority, soft injection, violations buried in normal paragraphs)

Four independent LLM generators (glm-5.3, kimi-k3.1-pro, deepseek-flash, MiniMax-M3.1-Flash) each produced the same 524-sample quota with self-checked labels; merging them found exactly one near-duplicate. The four-generator spread matters: no single generator's phrasing bias dominates the evaluation.

Method

Per model, 4,390 evaluations: every sample against every rule, with rules phrased in both English and Portuguese. We compare AUROC (threshold-free ranking quality), separation (mean P(yes) on violations versus benign; the property that makes a customer's 0.85 threshold meaningful), and recall at a 3% benign false-positive budget. Raw thresholds are not comparable across models; the operating curves are.

Quality results

AUROC per rule (range across question languages), with the worst recall at 3% benign FPR:

Two details deserve more attention than the ranking. First, the 27B's operating thresholds sit at 0.06–0.54 with recall above 0.97: customers get a wide, forgiving band instead of thresholds crammed at 0.95 on overlapping distributions. Second, d1-2B's confidential-request weakness is the model, not the quantization: full-precision bf16 lifts it only slightly over Q8 (worst recall 0.52 vs 0.24), so no serving fix rescues that rule.

The Portuguese result

On pt-BR content, worst-cell recall per model: clef 27B 0.97, jev 0.83, clef-flash 0.73, d1-2B 0.24. The open 27B led the proprietary reference on the market-relevant language, not just overall. Cross-lingual authoring also held for jev, clef, and d1: English-phrased rules covered pt-BR content within about 0.005 AUROC, so operators write rules once.

The contrast case was disqualifying rather than close: the English-only GLiFormer model scored AUROC 0.097 on pt-BR hostile content, confidently inverted, ranking insults as more benign than polite complaints. A model that is wrong with confidence is worse than one that is uncertain, and no threshold fixes that. If your traffic includes Portuguese, measure it explicitly; an English-only leaderboard will not warn you.

Latency is a serving-engine decision, not a model property

The most surprising measurement had nothing to do with accuracy. Liquid's card promises all questions answered in one forward pass. But llama.cpp evaluates d1-2B questions with per-question passes, so latency grows linearly with rule count, while Liquid's Transformers reference keeps it nearly flat. At 8 KB states with 16 rules that is a 29× difference: 74 ms versus 2,168 ms.

p50 latency at 16 rules per request:

The matrix points to three practical tiers: clef-flash 9B Q8 is the comfortable production fit on a 16 GB card; the 27B's quality ceiling needs a 24 GB card with Q4_K_M fully on GPU (our 16 GB run required hybrid partial offload); and the only in-budget CPU option is d1-2B Q8 via llama.cpp. One deployment pitfall: on Blackwell GPUs, llama.cpp silently linked Ubuntu's CUDA 12 runtime against a CUDA 13.3 compile and crashed clef with a cryptic SOFT_MAX error. Pin -DCUDAToolkit_ROOT to the matching toolkit. The issue is already reported upstream, and the fix is a one-line CMake flag rather than a source patch.

Off-the-shelf today, better with your own data tomorrow

One property of the d1-2B result deserves emphasis. Those numbers come from off-the-shelf weights with no fine-tuning of our own: no task-specific training, no calibration work beyond choosing thresholds. And d1-2B already has the best latency profile we measured: 7–74 ms across the matrix, every question answered in one forward pass.

Decision models in this family are designed to be customized on your own data, and Liquid offers fine-tuning support. The natural expectation is that domain fine-tuning moves exactly the cells where the off-the-shelf model is weakest: public-versus-confidential boundaries and subtle Portuguese injections are organizational dialects as much as general knowledge. We have not measured that yet. It is the next experiment, not a claim in this benchmark.

Limitations

The corpus is synthetic with generator-self-checked labels and no independent human annotation; four rules are a minimal guardrail set, not a safety taxonomy. All numbers are single-desktop, contended-environment measurements, and hosted jev latency was not compared. Treat the published thresholds as a starting point for your own calibration, never as defaults to copy.

Reproduce it

Everything needed to reproduce it (the corpus, the harness, the raw CSVs, the tests, and the exact serve commands) is being released in the public Chrysent-AI/ai-benchmarks repository, systemone-ptbr-guardrails directory (to be released soon). The formal nine-section study, with the full failure-case analysis, is at chrysent.ai/benchmarks/systemone-ptbr-guardrails.

What this powers at Chrysent

These measurements are how we chose the decision backend for Chrysent Touchstone, our content-governance product (private beta): customer-authored natural-language rules with customer-chosen probability thresholds, evaluated on prompts and model replies, in English and Portuguese, on your own infrastructure, without sending your traffic to a third-party evaluation API. If you run bilingual AI traffic and want to see these rules working on your own content, ask for a demo.

The same measurements feed the broader Chrysent AI Gateway story: a governed stack from prompt to tool execution. Content governance decides what a prompt or model reply may contain; the gateway enforces routing, policy, and provider controls around it: one governed path, not a pile of point integrations. Touchstone and the Gateway are both available in private beta; the demo form covers both.

And the improvement compounds. When an organization enables content retention, its gateway traffic and governance decisions stay inside its own infrastructure. Periodically retraining the guardrail model on that retained data (your real prompts, real near-misses, real attacks) means a classifier that improves on the distribution you actually serve, self-hosted, with your data never leaving your organization. That is the direction we are building toward: guardrails that get better on your traffic, not on someone else's benchmark.