TypeSafe Poc

Calibration matters more than raw accuracy here. If 85% confidence only hits 60% of the time, automation is unsafe; a well-calibrated moderate model can still auto-route most tickets through the confidence lanes.

At or above 0.80 it routes directly, 0.50 to 0.80 routes with light review, and below 0.50 defers to a human with the top two suggestions. All cutoffs are configurable through environment variables.

A ticket is high priority when urgency probability exceeds 0.70 or frustration reaches 4.0. The Outros option gives the model an explicit abstention so forced choices do not compress the distribution.

Run make eval-dry for a credential-free simulated pass, or point TYPESAFE_BASE_URL at a local mock. Live eval and make run need Go 1.24 plus TYPESAFE_API_KEY, serving POST /triage on port 8080.

The JSON Result carries the chosen department with its confidence and the margin between the top two options, the full probability vector across all six departments, an urgent flag with urgency_prob, the frustration level already on the 1 to 5 scale, the decision path of auto, review, human or degraded, a priority, latency_ms and token usage. Accepted fields are id, at least one of assunto or corpo, plus plano and interacoes_anteriores.

A proof of concept, by its own scope statement. It covers no custom model training, no fine-tuning and no integration with a real ticketing queue. The evaluation set is 50 synthetic tickets in testdata/tickets.jsonl, which the README calls too small and too artificial to inform product decisions, and says metrics on it tend to be inflated. Validate against real tickets before anything goes to production.

cmd/triage-server exposes POST /triage and cmd/triage-eval runs the offline evaluator; internal/triage holds the three questions in questions.go and the decision rule in decision.go, internal/httpapi validates the payload, internal/eval loads the labelled JSONL, and pkg/typesafe is a dependency-free client handling retry plus typed noul, choice and score parsing. The make targets are build, test, vet, fmt, lint and clean, all on Go 1.24.