Question 1

What is the main finding of this survey experiment?

Accepted Answer

Asking a yes/no survey item as a Noul (one probability) rather than a two-option Choice moved Jev's distribution gap from 0.1985 to 0.1530 -- further than the entire gap measured between Jev and GPT-4.1 on the same benchmark, at the same price.

Question 2

Was Jev better than GPT-4.1 overall?

Accepted Answer

It depends on the elicitation. Jev's Noul arm won all six reported measures at about a thirty-fourth of the cost, but GPT-4.1 probabilities were ahead on the per-column test fixed in advance, and GPT-4.1's hard answer was the most accurate per person.

Question 3

What was the experimental setup?

Accepted Answer

Jev 1.13.0 and GPT-4.1 played the same 300 Twin-2K-500 respondents across 108 questions, 16 behavioural-economics tasks and 24,596 answered cells per arm, in five arms that differ only in which model was asked and how. Each persona is built from that person's 14 demographic items, and the held-out behavioural-economics block is the exam; the author collects none of it and redistributes none of it.

Question 4

Why does the write-up lead with a distribution gap instead of accuracy?

Accepted Answer

Nobody can predict how one individual answers a trick question, and the humans themselves only reproduce their own earlier answer 81.68% of the time, with every arm scoring below a baseline that ignores the persona entirely. What a survey is for is the population number, so the deciding metrics are soft total variation distance on the 65 two-option columns and soft Wasserstein-1 on the 43 ordered ones.

Question 5

What is the mechanism behind the Noul result?

Accepted Answer

Saturation. Asked as a two-option Choice, Jev put probability exactly zero on one of the options in 16.6% of two-option cells; asked as a Noul it did so in 0.0%. On the same cell and the same label the two forms disagree by a mean of 0.13, so they are not variants of each other. The arm is 64.6% Noul and 35.4% Choice by cell, because a Noul cannot be asked on the multi-option columns.

Question 6

What caveats sit on the headline number?

Accepted Answer

The Noul arm was built after seeing the Choice result, so it cannot settle the comparison the tests were written for, and on the stricter per-column rule fixed in advance GPT-4.1 probabilities are still ahead at p=0.9995 because one task holds 40 of the 65 columns. Calibration missed its 0.05 bar for every arm at once, 0.1472 at best, and adding option descriptions changed only the 40 pricing columns, which is the control proving the run was clean. MIT licensed, self-funded work with no vendor involvement.

Question 7

What are the five arms and their actual numbers?

Accepted Answer

Jev Choice, the arm the comparison was set up around, sits at 0.1985 distribution gap and $4.01; adding option descriptions leaves it at 0.1985 for $4.03; Jev Noul reaches 0.1530 at $4.02; GPT-4.1 verbalized probabilities sit at 0.1789 for about $136; and the GPT-4.1 hard-answer arm is worst on distribution at 0.2037 yet best on accuracy at 69.32%. The descriptions arm changed only the 40 pricing columns, which is the control proving the run was clean.

Jev Synthetic Survey

About this project

Highlights

Questions about this entry

Related terms

More Jev resources like this

From the Jev community