Skip to content

Jev Guides & Articles

Compliance evidence: model cards, human oversight, and audits

What a decision-model deployment needs to show an auditor: model cards, human oversight, retention terms, and the limits of vendor evidence.
Published
Category
Jev Guides & Articles
Resources cited
11
Reading time
6 min
Tags#security#compliance#audits#legal

Compliance evidence is not a badge on a pricing page. It is a set of artifacts with dates, owners, scope, and a test that produced them. A model card that says "calibrated" is not evidence. A calibration report on your workload, with the version and the method, is. The distinction matters most when a procurement review asks how a decision model was selected and who signed off on the thresholds.

The contract stack

TypeSafe publishes the expected set under /legal. The legal hub is a thin index: it links the documents, states the no-training commitment, and offers zero data retention for enterprise through sales.

DocumentDateScope and what it proves
Terms of Use19 September 2026Website access only; a separate agreement governs the services
Master Customer Agreement23 September 2026Contract for services named in an order; license and resale limits
Data Processing Addendum24 April 2026Controller/processor roles; breach, subprocessor, and retention terms
Privacy Policy19 November 2025Personal-data categories; no training on prompts; no sale or sharing
Acceptable Use Policy23 September 2026Prohibited content and conduct; not exhaustive

Three details are worth flagging in an audit file. The site terms are not the API contract; the MCA covers the services named in an order, and a separate written agreement overrides it. The DPA controls to the extent of any conflict with the other terms, and its Schedule I states the subject matter, nature, purpose, data types, and data subjects. The privacy policy predates the September 2026 terms, so a product agreement can add privacy terms on top of it.

The DPA also commits to breach notification without undue delay and within 72 hours of awareness, and to reasonable advance notice before a new subprocessor gets access, with a 15-day objection window. Those are the clauses to quote when counsel asks what happens on a bad day.

Model and evaluation evidence

The hardest evidence to produce is independent accuracy. The early evidence audit is a useful corrective to vendor claims. It reviews 28 papers posted between 19 and 24 September 2026, nine days after Jev's release, and relates them to earlier work on label-probability classification, constrained decoding, reranking, calibration, and cascades.

Its finding is narrow and important: the typed readout has not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks. The audit derives a 14-item evaluation checklist from weaknesses that recur across the reviewed studies. The caveat is the corpus: nine days after one model's release is an early evidence map, not a settled assessment.

TypeSafe's workflow evals supply the vendor-side numbers. They cover four workflows: security incidents, agent trace observability, invoice processing, and customer service. Models are scored as workflows of Noul, Choice, and Score questions against consensus labels from GPT-6 Astra and Claude Fable 5.1 at high thinking. Jev averages 67.8% accuracy at $0.0004 and 0.4 seconds per case. The page also reports that every model is more accurate, cheaper, and faster in a workflow than with the same policy as a prompt.

Two caveats belong in the file. The evals are vendor-run, and the reference labels come from two frontier models; the harness and label correctness are assumed rather than independently audited. The "structure is always better" claim is an average across the four workflows, not a guarantee for yours. The benchmark-claims glossary entry covers how to read numbers like these.

Human oversight that can be audited

A human-oversight claim needs a mechanism: who sees the case, what they see, how long they have, and what happens if they do nothing.

For a decision model, the defensible pattern is a confidence gate with a named queue. TypeSafe's confidence documentation describes the mechanics: confidence is a 0-1 statistic derived from the distribution, Noul answers carry none, and thresholds depend on the domain. The docs suggest starting conservative and tuning against your own data, with an example that routes below 0.5 to a human and requires above 0.9 to auto-approve a transfer.

That is a starting point, not an audit artifact. The artifact is your threshold table, the workload it was tuned on, the measured false-positive and false-negative rates at that threshold, and the name of the person who approved it.

The DPA covers the vendor side of oversight: processing only on documented instructions, no sale or sharing, no retention beyond the instructions, and the 72-hour breach window. The enterprise page adds SOC 2 Type II, requestable through the Trust Center, and zero data retention for enterprise customers. It is a marketing page; the cost case studies at 88% and 98.7% lower cost have no methodology on that page, so do not treat them as audited results.

Accessibility and procurement

The accessibility statement is a good example of evidence with a scope limit. It states that WCAG 2.2 Level AA is the guide for improving typesafe.ai, not a completed conformance claim. It names one known gap: some content, including charts, does not yet have complete text descriptions. Problems go to [email protected]. It was last updated 11 October 2026, and it gives no audit date, testing vendor, or conformance report.

It also addresses typesafe.ai and does not name docs.typesafe.ai, console.typesafe.ai, or help.typesafe.ai. In an audit pack, record it as an open gap with an owner, not as a pass. If your procurement process requires a VPAT or an accessibility conformance report, this statement does not substitute for one.

The audit pack

A workable pack for a Jev deployment has eight artifacts.

ArtifactSourceWhat it answers
Signed MCA and orderTypeSafeWhich services and limits apply
Executed DPA with Schedule ITypeSafeRoles, data types, breach and subprocessor terms
Current privacy policy and AUPTypeSafeTraining, sale, and prohibited-use commitments
SOC 2 Type II reportTrust Center requestIndependent control testing, with period and scope
ZDR confirmationSales, in writingRetention for the specific surface you call
Model and question registryYour repoWhat was evaluated and what changed
Calibration and threshold reportYour evalsWhy the operating point is defensible
Incident and access-review recordsYour opsWho approved what, and when

The data-retention article covers the retention side in more detail, and the reproducibility article covers version pinning. The measuring article covers evaluation design.

Freshness, versioning, and what is missing

Legal documents change. The terms were updated on 19 September 2026, the MCA and AUP on 23 September, the DPA on 24 April, and the privacy policy on 19 November 2025. Keep a dated copy of each version you relied on, with a hash. A link to a page that may change is not a record.

Model versions move too. Pin the model ID, the question text, the option order, and the policy version, and re-run the evaluation when any of them changes. A floating alias makes last quarter's calibration report describe a different system.

What the vendor does not supply is the part your auditor will ask for: independent accuracy on your workload, an adversarial audit of your application path, and evidence that your thresholds were chosen before you saw the results. The evidence audit gives you a 14-item checklist to structure that work. Use the vendor documents for the contract facts, and build the performance evidence yourself.

From the Jev directory

The resources behind this article.

An audit of 28 papers posted between 19 and 24 September 2026, nine days after TypeSafe released Jev, relating their findings to earlier work and deriving a 14-item evaluation checklist. The typed readout shows no independent accuracy advantage yet; the clearest gains are latency and cost.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
TypeSafe's accessibility statement: the site is being improved against WCAG 2.2 Level AA, some charts still lack complete text descriptions, and problems can be reported to [email protected].
Jev Guides & ArticlesDocs#official#accessibility#legal
Official
How TypeSafe reports certainty, how it differs from probability, and how to use it architecturally to gate and route decisions.
Jev Practices & PatternsDocs#official#confidence#routing
Official
The enterprise page for Jev: two published case studies with 88% and 98.7% lower cost, SOC 2 Type II and zero data retention, a no-training-on-inputs commitment, and higher rate limits for teams.
Jev Guides & ArticlesDocs#official#enterprise#compliance
Official
The hub for TypeSafe's legal and policy documents, plus the data commitments attached to the API.
Jev Guides & ArticlesDocs#official#docs#privacy
Official
Rules for using TypeSafe services, last updated 23 September 2026. They ban illegal content, rights violations, and phishing or spoofing.
Jev Guides & ArticlesDocs#official#legal
Official
TypeSafe's data processing addendum, last updated 24 April 2026. For customer personal data, the customer is the controller and TypeSafe is the processor.
Jev Guides & ArticlesDocs#official#legal#privacy
Official
TypeSafe AI, Inc.'s master customer agreement, last updated 23 September 2026. It is the contract for services named in an order.
Jev Guides & ArticlesDocs#official#legal
Official
TypeSafe's privacy policy, last updated 19 November 2025. It covers the website, the playground, and APIs that link to the policy.
Jev Guides & ArticlesDocs#official#legal#privacy
Official
TypeSafe's website terms, last updated 19 September 2026. They cover typesafe.ai. A separate agreement covers the products and services.
Jev Guides & ArticlesDocs#official#legal
Official
Published evaluations of four automation workflows, security incidents, agent trace observability, invoice processing, and customer service, comparing Jev and frontier LLMs as structured workflows versus single prompts.
Jev Practices & PatternsDocs#official#benchmarks#evaluation
Official
Back to all articles

Related Jev articles

How to design a decision pipeline that minimizes personal data, what zero data retention does and does not cover, and what to log for replay.
Jev Guides & Articles#security#privacy#retention
Read article
The typed-answer shape removes a category of data risk, not all of it. Zero data retention per surface, the legal set to read, and the controls that remain yours to run.
Jev Guides & Articles#privacy#compliance#legal
Read article

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai