A tool call is trusted twice: once when the agent decides to make it, and again when the application believes what comes back. Most agent code validates neither boundary properly. The schema checks the shape, the model checks the meaning, and the result is accepted because it parsed. That is how a correct tool choice turns into a bad action, and how a clean JSON response turns into a wrong decision.
Two validation boundaries
The first boundary is outgoing. The agent has to pick a tool from a catalog and fill its arguments. Those are separate skills, and they fail differently. Candidate selection fails when the right tool is present but ranked below the cutoff, or when a near-miss tool looks more relevant. Argument construction fails when a required field is missing, a value is invented, or a string is copied from the wrong part of the request.
The second boundary is incoming. A tool result can be well-formed and still be stale, empty, truncated, injected, or semantically wrong. The decision layer has to validate the envelope before it reasons about the content.
| Boundary | What to check | Typical failure | Deterministic guard |
|---|---|---|---|
| Candidate selection | Is the right tool in the offered set? | Near-miss tool wins | Retrieval prefilter, candidate cap |
| Argument construction | Types, enums, required fields, provenance | Invented value or wrong slot | Schema validation, exact-substring extraction |
| Policy | Is this call permitted at all? | Model confidence used as permission | Deny-by-default rules before the model |
| Response envelope | IDs, finite probabilities, option membership, usage | Malformed or mismatched answer | Typed rejection, retry or escalate |
| Result content | Is the evidence fresh and plausible? | Stale or injected output | Freshness checks, source rules |
The ordering is not cosmetic: a policy denial should happen before a model call, a malformed answer before a branch, and a stale result before it becomes evidence.
Choosing the tool is a decision
Tool search is a retrieval problem until it is not. Jev Tool Search measured four approaches over 525 real MCP tools from 69 servers and 94 human-annotated tasks. BM25 found the right tool first 32% of the time, Voyage embeddings 46%, a two-stage Jev search 56%, and Voyage rerank-2.5 60%. The experimental Jev engine with an embeddings prefilter reached 59% with 92% in the top five, at $0.26 per 1,000 searches versus $2.23 for the reranker and $1.03 for two-stage Jev. With 94 tasks, the README warns that differences under about 10 points are noise, and rows come from different runs.
That caveat is the useful part: a four-point win is not a reason to rebuild the router, and expensive is not automatically better. Measure on your own catalog.
Jev Toolspace asks a different question: instead of picking one tool, score every tool's relevance so only the relevant subset keeps full documentation. One API call asks a yes/no Noul question per tool and returns independent probabilities, so tool scores do not compete. With a Qwen2.5-7B executor, its reported TGB macro accuracy is 80.7 for the trace-based setting against 69.5 keep-all, and BFCL multi-turn is 17.3 against 11.3 keep-all and 13.8 Tool2Vec. Those are the project's own runs on its own protocol, with one executor model and menus padded to 30 tools: a research report, not a general benchmark.
Soro-Kev-4B occupies the local-model end. It reads one state document plus typed questions and returns calibrated tool-routing probabilities in one forward pass. The card reports 89.00% zero-shot tool routing on its Universal Tool Harness and 94.50% on external human holdouts, with a median prefill of about 413 ms on an RTX 4050 Laptop GPU or Kaggle T4. A learned temperature of 1.62 and a split-conformal quantile of 0.8066 at 90% coverage drive EXECUTE, ASK, and abstain outputs. The numbers are self-reported with no third-party reproduction, and training states were capped at 4,096 tokens even though the card advertises 32,768-token states.
Arguments need a schema and a policy floor
Argument validation starts with the schema and ends with provenance. jev-mcp-dispatcher is instructive because it refuses to generate values at all: non-enum arguments use a per-word Noul pass, and the selected values are exact substrings copied from the command. That rules out paraphrase and invented text, but it also means typos and pronouns are not resolved, and two same-typed parameters can claim the same word. The dispatcher supports only string, integer, number, boolean, and enum parameters, and optional parameters are still force-extracted. It is a proof of concept, but the provenance rule is right.
The policy floor should be deterministic. jev-tool-permissions checks allow and deny rules before anything reaches Jev; deny rules run first, and the model is never consulted for them. If the Jev call errors or times out, the gate defaults to ask-human, and tool pruning keeps every definition rather than silently dropping something relevant. MCP Router applies the same principle at the catalog level: model scores are relevance data that can never widen access, and a deterministic deny-by-default policy engine gates execution.
jev-guard adds the pre-execution check for coding agents. It decides credential files and workspace escapes locally in under a millisecond, sends ambiguous or destructive calls to the model, and fails safe by asking when the API is unreachable. The fast-path claim is the project's own; no independent benchmarks are published.
Do not ask the model to count or compare dates
The jaggedness list is the best short checklist for argument and result validation on jev-1.13. Counting is unreliable and error grows with size, so the guardrail is one Noul per item with the sum done in code. Dates are read as text, so extract components with Choice and compare them outside the model. Score levels are weak at numeric calibration, so do not interpolate an exact magnitude between levels. Structural invariants do not hold: complementary probabilities can sum past 1, and the same question can disagree across primitives.
These are not reasons to avoid the model. They are reasons to put arithmetic, ordering, and invariant checks in code.
Results are evidence, not truth
A tool result is an observation, and observations need provenance. If a web page or file can contain instructions, the result is also an input channel for injection. Confidence-gated routing makes the related point about answers: confidence is not permission, and a high-confidence answer can still be wrong. Keep hard limits next to the model for irreversible actions.
A practical result validator checks four things: the answer IDs match the questions asked, every probability is finite and within [0,1], every choice is one of the offered options, and the model and usage fields are present. On failure, return a typed error and let the caller retry, escalate, or fall back. Never coerce a malformed answer into a plausible one.
A two-sided pipeline
- Build the candidate set deterministically, then let the model rank within it.
- Validate arguments against the schema and against the user's request, not just the type.
- Run deny rules and budgets before the model is called.
- Validate the answer envelope, then validate the content's freshness and source.
- Keep a human path for irreversible actions and a typed failure for everything else.
- Test the validator with malformed distributions, missing IDs, extra options, and stale results.
The function-calling layer is where these two boundaries meet: the model proposes, code disposes. For the state design that makes arguments easier to validate, see Designing the State; for instructions and criteria, Structured Instructions for Typed Decisions; and for the failure catalogue, Designing Around Jagged Edges. A confidence score is a useful input to a threshold, never a substitute for validation.
