Will enabling this ship my production traces to a third party?

The judged text goes to TypeSafe, in one POST carrying every question about that text. Source comments say customer provider keys are not sent on that path and that LangWatch's own JEV_API_KEY is used, but the trace content itself does leave your machine.

What is the catch with 32k tokens of judged text per call?

The cap. Source comments measure about 250 ms per call with 32k tokens of judged text inside a 64k limit, billed on input only. Longer traces must be chunked before judging, and the README does not document how chunk boundaries affect scores.

Do I need one API call per question, or can I batch them?

One call per text, not per question. The client packs every Noul, Score and Choice question about that text into a single POST. That is the batching win, and it also means you cannot easily split the questions across separate calls.

Is this locked to the hosted platform, or can I self-host the Jev path?

Self-hosting is documented, with only Node.js required. The core is Apache 2.0, and the Jev classifier sits in the open-core tree, while modules under platform/app/ee/ need a commercial license for production use.

What license is LangWatch under?

The core is Apache 2.0, the SDKs are MIT, and enterprise modules under platform/app/ee/ need a commercial license in production; it is GDPR compliant and ISO 27001 certified.

What key does the Jev instant-evals path use?

LangWatch's own JEV_API_KEY; source comments say customer provider keys are not sent on that path, even though the judged trace text itself goes to TypeSafe.

What does LangWatch do besides instant evals?

It covers LLM ops observability, agent testing, evaluations and prompt management, coding-agent tracking with cost per PR, an AI gateway with virtual keys and budgets, and AI governance.

CommunityJev Benchmarks & Evaluations

LangWatch

LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.

See highlights

Open github.com

Stars: 4.9k
Language: TypeScript
License: APACHE-2.0
Published by: Community
Author: langwatch
Last verified: Oct 2026

langwatch/langwatch

Organization repository on GitHub

View on GitHub

Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.

Forks: 409
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

LangWatch is the open-source platform for AI in production: trace, test, route and govern every LLM call in your company, from the agents you build to the coding assistants your engineers use. LangWatch is Apache 2.0. View docs. Only Node.js required. For production, see Self-hosting LangWatch. LLM Ops: Observability · Agent Testing · Evaluations · Prompt Management. Coding Agents: sessions, cost per pull request and per team, privacy controls.

Quoted from the project's README. Read the full README.

Tagscommunity typescript evaluation observability

Works forJev for Evaluation Jev for Classification Jev for Scoring

Used inJev for Software & DevTools Jev for Data & Analytics

Languages: TypeScript
Platforms: API
Capabilities: Open sourceRequires API key

Highlights

platform/app instant-evals classifier jev.client.ts documents one POST carrying every question about a text.
Comments measured about 250 ms per call, 32k tokens of judged text inside a 64k cap, and input-only billing.
Uses LangWatch's JEV_API_KEY; customer keys are not sent on that path per the source comments.

Watch out

Apache-2.0 core with enterprise modules under platform/app/ee/ needing a commercial license in production.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

Precision and recall

More Jev resources like this

OpenWork

24k

An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.

Jev Benchmarks & Evaluations#community#typescript#agents

Community

different-ai/openworkTypeScript

jev vs. open alternatives

Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.

Jev Benchmarks & Evaluations#community#evaluation#document-processing

Community

run-llama/jev_vs_ossJupyter Notebook

DeepSearcher Jev stopping experiment

8.3k

Agentic search evaluation inside Zilliz DeepSearcher: a standalone harness replaces the ChainOfRAG reflection call with one Jev Noul per round to decide whether to stop within a seven-round budget, then compares stopping behavior, evidence recall and decision cost on 100 sampled 2WikiMultiHopQA queries.

Jev Benchmarks & Evaluations#community#rag#evaluation

Community

zilliztech/deep-searcherPython

hev reranker

A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.

Jev Benchmarks & Evaluations#community#evaluation#python

Community

hev/rerankerPython

jev-acento

Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.

Jev Benchmarks & Evaluations#community#evaluation#calibration

Community