Jev Ids

At five examples Jev matched LLM precision while running faster and cheaper per flow, but the LLM led on overall F1 because it caught more attacks of kinds shown in the examples. Jev led on zero-day kinds absent from the examples.

The forest is trained on the same few examples, so with tiny example counts it overfires on benign traffic. On the same benign flows Jev produced far fewer false alarms in the reported runs.

Same flows, same examples, same 0.5 cut go to Jev, an LLM agent baseline, a Random Forest, and an Isolation Forest on benign traffic alone. A compare command pairs runs flow by flow with a McNemar test.

Yes to both, and the README does not document a subset path. Download NSL-KDD into data/raw/nsl-kdd/, run scripts.prepare_nsl_kdd once, then run the smoke split with five flows. TYPESAFE_API_KEY covers Jev alone; the Gemini baseline needs gcloud application-default login plus GOOGLE_GENAI_USE_VERTEXAI and the project settings, and DEEPSEEK_API_KEY or CHATGPT_CLIENT_ID only for the other LLMs. Each run writes config.json, per-flow prediction rows and raw answers for inspection.

Clone the repo, `uv sync`, set TYPESAFE_API_KEY, download NSL-KDD into data/raw/nsl-kdd/ and prepare it with `uv run python -m scripts.prepare_nsl_kdd`. Then `uv run jev-ids run --dataset data/nsl-kdd/dataset.json --detector jev --split smoke --k 0,1` runs the five-flow smoke split.

One network flow goes into the state next to the instructions, column names, five category descriptions and labeled examples, and two questions are asked: is_attack returns a probability, and category returns one of five options with a confidence. The verdict is attack when the probability reaches 0.5.

Offline, as tokens times the list prices in prices.json, for every detector alike. Jev charges $0.042 per million input tokens and nothing for output; at about 1,800 tokens per flow, a million verdicts cost about $74.