A decision cache sounds like a contradiction. If two inputs are different, the answer may be different, and serving the first answer for the second is how a cache becomes a bug. But most decision traffic is not unique. It is the same ticket reopened, the same document reclassified after a field changed, or the same policy question asked in five branches of one workflow.
The useful move is to separate the kinds of repetition before choosing a cache.
Three repeats worth separating
The first is an exact repeat: the same state, the same questions, the same model version. Hashing that tuple is cheap and safe, and the cache hit is not a model decision at all.
The second is a same-intent paraphrase: a retried agent turn, a reworded support message, or the same request with a different greeting. A string hash misses it; an embedding cache may match it too loosely. This is the case that needs an adjudicator.
The third is repeated computation inside one request. If two branches both need urgency and refund risk, asking both questions once against one state is cheaper than two requests and avoids the cache entirely. The speculative fan-out pattern is the design version of this idea: ask everything the workflow might need, then ignore the answers the taken branch does not use.
Exact caching is a hash, not a model
For exact repeats, build the key from the full request: state bytes, question ids and wording, criteria, model version, and any instructions. Then store the answer with a timestamp. No model call, no ambiguity.
This is also the right place to put idempotency. The LLM cost optimizer records idempotency keys alongside its cost provenance and limits itself to at most two attempts. That combination (a key that identifies the logical request and an attempt counter that does not multiply the work) is what keeps a retry storm from becoming a billing incident.
The API reference helps here because answers come back keyed by the ids you choose, and those ids are never sent to the model. You can keep internal cache keys and question names inside your system.
Semantic caching needs an adjudicator
A cosine-similarity cache answers a different question: are these two inputs close enough in embedding space? That is a retrieval heuristic, not a decision. Two tickets about the same product can have nearly identical embeddings and opposite urgency.
jevcache takes the more defensible route. It sits in front of an OpenAI-compatible endpoint and asks a Jev or local System One adjudicator whether a paraphrase has the same intent, with a freshness check and a time bound. It fails open on adjudicator errors, and the repository reports zero false positives on a live 100-pair run against a Jaccard baseline, while explicitly treating that as precision-first practice, not a guarantee. The caveat matters: personalized content, tool-call replays, and multi-tenant use without a tenant header are out of scope.
GPTCache is the older, more general version of the idea. It supports exact and similarity matching, and its TypeSafe integration uses Noul checks to judge whether a cached response is still fresh for the current query. The project is MIT-licensed and widely used, but its README warns that the API may change and that new model adapters are no longer being added. Treat it as infrastructure you may need to fork, not a turnkey policy layer.
Deduplicate inside the request first
Before adding a cache, check whether you are paying twice for the same state; parallel questions are the structural fix. The parallel questions cookbook ran 13 questions (8 Noul, 2 Choice, 3 Score) over a 54,000-character document. One batched call cost $0.000497 and took 0.27 s; 13 single-question calls cost $0.006090 and 2.71 s. That is 12.2x cheaper and 10x faster, with the same answers across five repeats.
The mechanism is simple: the state is billed once, not once per question. The fan-out benchmark measured the same effect across 2,976 requests: median input-token savings at eight questions were 76% on a tiny control, 77% on a small state, 84% on a medium state, and 86% on a large state. Every request also carried about 261 input tokens of fixed overhead, which is the floor you pay even for a one-word state.
If your workflow sends five separate calls because the code is organized around five functions, the first optimization is not a cache. It is one request with five questions.
Freshness, tenancy, and versioning
A cache key that omits the model version is a time bomb. The models page shows that jev-latest and jev-preview both pointed at jev-1.13.0 at capture time, and that aliases can move. A cached answer produced by 1.13.0 may not be the answer 1.14.0 would give, especially on multilingual input where the release notes expect the largest gain.
The key also needs a tenant boundary. A cache shared across customers can leak one customer's phrasing into another's decision, even if the stored value is just a probability. If you cannot isolate the cache, isolate the key.
jev-workbench offers a useful model for the versioning problem: publishing a judgment function freezes the config and model version, and grants pin a client token to specific function versions. A published v2 does not silently upgrade callers. You can copy the principle without adopting the tool.
What a cache cannot fix
A cache does not improve a bad question. It makes the same bad answer cheaper. The workflow evals put Jev at 67.8% average accuracy at $0.0004 per case across four workflows, against 74.1% for a frontier model at $0.0836 and 23.3 s in the independent audit. A cache can cut the $0.0004; it cannot close the accuracy gap on invoice processing, where Jev scored 61.8% against 79.1% in that audit.
A cache also does not replace a confidence policy. The confidence docs recommend routing low-confidence answers to a human or a fallback, and a cached high-confidence answer from last month is not the same evidence as a fresh one. If the decision is consequential, cache the inputs and the model call, but re-run the policy.
The Jev inbox queue is a good small example of caching used correctly. It answers seven typed questions per email thread, keeps the policy in code, and caches answers so changing a threshold does not require new API calls. Its demo sorted 47 hand-labelled emails in 1.4 s with 0 of 23 actionable emails missed, and a real 100-email inbox in 2.7 s. The author is explicit that the demo labels are his own and that your inbox is the real test.
A cache policy you can defend
Write down four rules. Exact repeats use a hash key that includes model version, question text, and criteria. Semantic matches require an adjudicated same-intent verdict and a freshness window, and they fail open only if a stale answer is safer than no answer. Tenant and sensitivity labels are part of the key, not metadata added later. And every cache hit is logged with the key, the age, and the model version that produced it, so a wrong answer can be traced to the stored decision rather than to the current request.
Then measure hit rate and error rate together. A cache that saves 30% of calls but raises the error rate by one point is not a saving; it is a quality regression with a smaller invoice. The cost of a decision and one request, many questions explain why the input side is the right place to optimize. The reproducibility article explains why the key has to include the version.
