SystemOne decision models for bilingual (EN/pt-BR) guardrails

1. Question / hypothesis

Can customer-authored natural-language governance rules — a yes/no question plus a probability threshold — be evaluated reliably on mixed English and Brazilian Portuguese traffic by self-hosted open-weights decision models? The hypothesis was that open models would trail the hosted proprietary reference; the measurements inverted that expectation, with one narrow exception noted in section 6: jev retains a hostile-tone AUROC edge of 0.999 versus 0.997–0.998.

2. Product / version / date

Model-selection research supporting Chrysent Touchstone (private beta) natural-language governance rules. Study version 1.0.0, published 2026-11-04. Models under test: Cloudflare clef (27B) and clef-flash (9B), LiquidAI d1-2B, knowledgator GLiFormer-large via the jeff server, and the hosted TypeSafe jev-1.14.0 reference. All are SystemOne-API decision models.

3. Environment and dataset

Local desktop: NVIDIA RTX 5070 Ti (16 GB VRAM), 32-thread x86 CPU, contended desktop environment (not isolated-lab measurements). Corpus: 2,195 synthetic bilingual samples — 800 benign (456 near-misses) and 1,395 violations across four rules (pii_request, prompt_injection, hostile_tone, confidential_request), 1,046 English / 1,047 pt-BR / 102 code-switched, with a plain → subtle difficulty ladder. 100 hand-written samples plus four independent LLM generators (glm-5.3, kimi-k3.1-pro, deepseek-flash, MiniMax-M3.1-Flash), merged with exact + shingle near-duplicate removal.

4. Baseline

The hosted TypeSafe jev API (jev-1.14.0) is the proprietary reference implementation of the SystemOne decision API. Open models were measured as self-hosted alternatives on identical questions, corpus, and thresholds procedure. The English-only GLiFormer encoder (jeff) is included as a contrast case and was disqualified for pt-BR traffic.

5. Procedure and number of runs

Quality: every model answered all four rule questions in both phrasings for all 2,195 samples (4,390 evaluations per model; identical noul yes/no questions, probabilities recorded per sample and rule). Metrics: AUROC per rule × question language, separation (mean P(yes) on violations vs benign), and recall at the smallest threshold with benign false-positive rate ≤ 3%, split by content language. Latency: 200 B / 2 KB / 8 KB states × 1 / 4 / 16 rules, warmup plus 10–30 timed repetitions per cell, client-side round-trip (d1-Transformers runs are in-process). Raw threshold values are not comparable across models; compare AUROC, separation, and recall at matched false-positive budget.

6. Median / range and failure cases

AUROC per rule across English- and Portuguese-phrased questions, and worst-cell recall at 3% benign FPR. Full per-sample data is in the public repository.

Failure cases are concentrated, not uniform: d1-2B cannot separate public-financial near-misses from confidential-information requests (benign P ≈ 0.20–0.26 against violations at 0.83; worst PT recall 0.24), and its PT prompt-injection recall drops to 0.53–0.59. The English-only GLiFormer model inverted on pt-BR hostile content (AUROC 0.097 — confidently wrong), which is a disqualifying failure mode for bilingual deployment rather than a tuning problem.

On Brazilian Portuguese content specifically, worst-cell recall was 0.97 (clef 27B), 0.83 (jev), 0.73 (clef-flash), and 0.24 (d1-2B): the quantized open 27B led the proprietary reference on the market-relevant language. Cross-lingual rule authoring held for jev, clef, and d1 — English-phrased rules cover pt-BR content within ~0.005 AUROC.

All measurements in this study used off-the-shelf weights with no fine-tuning of our own. d1-2B's weakest cells (confidential near-misses, subtle pt-BR injections) are exactly the domain boundaries that organization-specific fine-tuning is expected to move; a periodic retraining loop on retained gateway and governance data — data that never leaves customer infrastructure — is future work, not a measured result here.

Latency (p50 client round-trip, 16-rule requests) shows the serving engine matters as much as the model: llama.cpp evaluates d1-2B questions with per-question passes, while Liquid's Transformers reference answers all questions in one forward pass — 29× faster at 8 KB states with 16 rules (74 ms vs 2,168 ms).

7. Limitations

Synthetic corpus with LLM-generated, generator-self-checked labels and no independent human annotation; four rules are a minimal guardrail set, not a complete safety taxonomy. Single desktop environment with contended measurements; absolute latencies are hardware-specific. Hosted jev latency was not measured and is not comparable to local engines. The 27B configuration on 16 GB is a hybrid partial-offload posture; its production home is a 24 GB-class GPU. An earlier 100-sample corpus produced perfect scores that did not survive the 22× difficulty-scaled corpus — small-corpus results should be treated as optimistic.

8. Reproduction guidance

The repository (artifacts to be released soon) carries exact llama.cpp serve commands (including the Blackwell CUDA toolkit pinning that avoids a known silent CUDA-12/13 linkage crash), the Transformers reference path for d1-2B, and the jev API runner (bring your own key). Quality runs regenerate identical CSVs on any hardware; thresholds must be recalibrated on your own traffic before production use.

9. Downloadable or inspectable artifacts

the public Chrysent-AI/ai-benchmarks repository, systemone-ptbr-guardrails directory (benchmark data, harness, and corpus to be released soon): corpus-v2.json (2,195 labeled samples), the SystemOne harness and hosted-API runner, raw per-sample CSVs for all five models across seven serving configurations, the written report, corpus-integrity and metric tests, and the earlier 100-sample results used for the difficulty-scaling comparison.

Related

See the product page for capabilities, supported systems, deployment, and security details. For private run reports under NDA, use the design partner contact form.