Jev within 5 points of a fine-tune

Not according to the person who tested it. The author's conclusion is explicit: a fine-tuned Qwen still beats it on latency by roughly 3x and comes very close on accuracy, so Jev is not yet a drop-in replacement for fine-tuned models on that task.

An internal one, and the post says nothing about it. There is no dataset, no task description, no label balance, and no threshold definition beyond the phrase matched precision, so the roughly 5-point recall gap cannot be reproduced or even scoped by a reader.

From the author's own full volume on early access, with sub-second latency. The post gives no request count, input size, or pricing assumption behind it, and it predates any public pricing you could check it against, so treat the number as one team's estimate.

It depends on whether your loop is latency-bound or accuracy-bound. The author valued the cheap zero-shot alternative enough to publish the comparison, but also concluded a fine-tune still wins on their task. A reply argues fine-tuning keeps beating general models on narrow tasks.

iden, who says he is building openmarket_xyz with an ML and systems background at Cambridge, testing on early access. That makes it a practitioner's private benchmark rather than a vendor evaluation or a third-party audit, which is why the missing dataset details matter more than usual.

That Jev was pointed at the task with no task-specific training, while the comparison model was a fine-tune the team had already built. The reported gap, within about five points of recall at matched precision, is therefore an untrained-versus-trained comparison, which is the more striking half of the result.

Skepticism toward general models. James replies that fine-tuning is still vastly underrated and that he does not see a future where a general model beats a narrow fine-tune on its specific task, especially on cost. The author had already reached a similar conclusion for their own task, so the reply reinforces rather than disputes the result.