Voice is a deadline problem. A turn detector has to decide whether the caller has finished speaking while the caller is still waiting, and an intent router has to choose a branch before the pause becomes awkward. The model is only useful if its decision fits inside that window.
The projects below cover turn detection, routing, rescoring, captions, and physical actions.
Turn detection is the first gate
Veris's turn-detection test asks Jev to label the caller's turn as complete, short (cut off mid-phrase), or long (paused for something longer). Jev gets the caller's words so far plus the agent's last line. Across 600 simulated calls (100 tasks times three runs in each of two configurations), interruptions fell from 52% to 11% of caller turns: clean 31% to 8%, street 28% to 9%, cafe 33% to 8%, and TV 64% to 14%, all significant at p below 10⁻⁷.
Task completion barely moved: the plain 0.8-second silence timer passed 50 of 300 calls, and the Jev-gated agent passed 53 of 300. Median reply time rose from 2.1 seconds to 4.6 seconds, mostly from held-open turns, while Jev labels returned in about 150 ms. TV noise remains the worst condition because the speech recognizer still writes TV speech down as the caller's, and Jev often waits for it to finish.
Veris is explicit that this is not a fair turn-detector comparison: one Pipecat agent, Pipecat 1.12 with Jev against 1.8 with a plain timer, and hold times the authors chose. Read it as a field test of the gate, not a leaderboard.
The voice turn-end demo is the lightweight version. It waits 0.5 seconds after speech ends, asks Jev whether the turn is really over, and maps the score to extra hold time: timeout=3.0 in the recorded output. Jev's processing time was logged at 224 ms. There is no published code, test set, or accuracy number. It is a prototype, not evidence.
Intent routing inside the pause
Jev Assist is a Home Assistant conversation agent that makes one system_one call per utterance on jev-latest, with retries and a 10-second timeout. The state is a JSON object with the utterance, language, entities, and areas. The router returns one of three kinds: fast_service, grok, or reject.
The fast path covers lights, climate, and covers. Multi-area requests such as all lights in the bedroom and hallway stay a fast-service union over the named rooms. Whole-home safety is enforced in code: target_area=none never fires all exposed entities of a domain, requiring a name match, an explicit area, or a single exposed entity instead.
The thresholds are the interesting part: FAST_MIN_CONFIDENCE=0.80, NOUL_YES_THRESHOLD=0.55, and NOUL_UNSURE_LOW=0.40. An uncertain fast-path match is rejected or handed to the Grok agent rather than firing a service. The integration ships no speech stack; it is a router in front of a conversation agent. Router tests mock Jev, questions are German and English only, and there is no license.
Confidence per action in the home
Jev Home Assistant Sentinel takes the safety boundary further. A bounded case goes to a decision model, a deterministic policy check runs, and state verification is recorded separately from the command result. The recommendation stays advisory rather than directly actuating devices.
Four decision modes cover hosted, local, and two fallback chains. It reaches hosted Jev over TypeSafe or OpenRouter, Clef on Cloudflare Workers AI, and open-weight Laya locally. local_model selects the checkpoint behind the same /v1/systemone shape without code changes. It ships as a HACS custom integration for Home Assistant, requires Python 3.10+, and publishes no benchmark or latency numbers.
The separation of state verification from command result is the pattern worth copying. A model's recommendation should never silently become the record of what happened. That is the same distinction the confidence-gated routing pattern makes between the answer and the permission to act.
Rescoring speech without a GPU
The speech-neuroprosthesis rescoring study poses rescoring as one typed decision that returns a probability for every candidate sentence. On 978 held-out sentences from a participant with ALS, the published decoder alone reaches 8.1% word error. Jev reaches 7.5%, against 7.8% for both OPT-6.7b and Qwen2.5-7B. With the decoder weight re-tuned, Jev reaches 6.9% against 7.2% and 7.4%.
The cost is $0.07 per thousand sentences with no GPU. End-to-end latency over the internet is 262 ms, of which 62 ms is at the provider; a local 7B model on a GPU is faster at 27 ms. The margins are small, the measurements cover one participant, and it is an arXiv preprint, not a product benchmark. The useful idea is architectural: keep the decoder's own score, add a model judgment, and re-tune the combination rather than replacing one with the other.
Qwen3-ASR-0.6B-Jev-Dutch is a different approach: a fine-tune of Qwen3-ASR-0.6B with a Jev head, 782,502,987 BF16 parameters in a single 1.57 GB safetensors file. It is tagged for Dutch audio classification, data curation, speaker attributes, and audio quality. It loads through custom code (modeling_jev.JevModel), has no hosted endpoint, and is Apache-2.0.
Live captions: annotate sentences, not streams
semantic-live-caption pairs append-only ASR with per-sentence Jev annotation for key points, emotion, and intent. Live labels are debounced and rate-limited, then the closed-sentence final judgment overwrites them; the ASR text itself never rewrites, so annotations can stick to sentences safely.
Intent labels pass four gates: a Choice hit, confidence at or above 0.55, a lead margin of at least 0.12, and a Noul recheck. High-false-positive categories like objection require 0.70. The page is a single dependency-free HTML file with keys in localStorage via a same-origin proxy. The ASR side needs an NVIDIA GPU server with vLLM and VAD weights, and Chinese calibration is weaker than English, so low-confidence Chinese labels are skipped.
This is the right way to put a model in a stream: make each decision about a bounded unit, keep the transcript append-only, and let the final label overwrite the provisional one. The per-tick and multimodal articles cover the same discipline.
Audio actions need an audit trail
jev-audio-beeper flags Spanish profanity by asking two typed questions per word in context: isInsult (Boolean) and severity (Score with four ordered levels). The pipeline runs ASR timestamps through parallel Jev evaluations, then threshold padding and merging, then an ffmpeg beep overlay that keeps the original duration.
The design is honest about the hard part: a false positive destroys audio. The default 0.72 threshold is described as a proof-of-concept starting point, not production calibration, and the README recommends hysteresis so a lower bar extends an adjacent span without starting a new one. The demo handles whole files; streaming is a 300–600 ms PCM ring-buffer sketch. There is no license file, and audio and transcript text are sent to the configured providers.
Keep the deadline in the architecture
Two numbers decide whether a voice design works: the p95 latency of each decision and the cost of being wrong at that latency. The confidence page separates the two. Choice and Score answers carry a confidence statistic; Noul answers carry only a probability. Thresholds should be set per action, and confidence is not permission.
For visual context, the Box question adds another deadline-sensitive decision. A Box answer is four edges as fractions of the image plus confidence, available on jev-1.14 with the images header. It has no not-found value, so it must be paired with a Noul that checks whether the target is present. Low-confidence boxes need a re-ask or a padded crop.
The latency percentile that matters is not the median. A turn gate that returns in 150 ms usually but 600 ms at p95 will produce a different conversation than the median suggests. Measure the tail, set the confidence gate per action, and keep a deterministic fallback for the turn you cannot classify in time.
