Q: Does Jev's stated confidence track its accuracy here?

Yes. Accuracy rises with stated confidence: 51.7% in the 0.50-0.60 band, 60.8% at 0.60-0.70, 67.9% at 0.70-0.80 and 78.5% at 0.80 and above. That lets you discard the bottom band and keep an edge worth having.

Q: How does Jev compare with common headline rules of thumb?

Jev scored 64.5% across 10,984 experiments, ahead of has-a-number at 58.5%, longer-headline at 54.2%, has-a-quotation at 51.9% and says-you-or-your at 50.3%. Shouted words won 41.6% of the time and a question mark only 35.5%.

Question 1

Are the accuracy numbers inflated by picking the easiest headline from each test?

Accepted Answer

They are easier than the general task, and the README says so. Each pair is the best and worst variant inside a single Upworthy test, selected to maximise the real difference, which is not the same as choosing between two arbitrary headlines. That selection is also why the z statistic on a chosen pair overstates its own significance; the author uses z only to bucket pairs into strata and deliberately quotes no p-value from it. A test only qualifies if every variant shared one eyecatcher_id, which cuts the 32,487-test archive to 13,279 headline-only tests.

Question 2

Does the model just prefer whichever headline is shown second?

Accepted Answer

It does, and that is measured rather than assumed. Across the sample Jev gives the first slot a probability of 0.428, not 0.500, so every pair is scored twice, once in each order, and the two are averaged. A test named for the effect proves the arithmetic: a model that always picked slot B scores exactly chance after averaging rather than a win. Without the swap, the reported number would carry a seven-point positional preference.

Question 3

Is a question mark really that damaging in a headline?

Accepted Answer

On this archive, yes, and the effect is large: headlines containing a question mark won 35.5% of the time across 2,624 tests, meaning they lost roughly two times out of three. Addressing the reader as you or your was worth nothing at 50.3% over 3,033 tests, a shouted word lost at 41.6% over 860, and having a number won 58.5% over 1,969. The README's own caveat is that this describes Upworthy's readers in that period, not a general rule about language.

Question 4

Can the model just have memorised the answers from the public archive?

Accepted Answer

The README argues memorisation is not a plausible mechanism here: Upworthy's headlines were public but the click counts were not until the archive was released, and reconstructing which of two variants won from 150,000 headlines is not something the author considers achievable. The available check is the split: the exploratory and confirmatory halves were released at the same time and score the same, 63.9% and 64.5%, with every design decision made on the exploratory side before the confirmatory side was run once.

Question 5

What is the placebo result in the headline benchmark?

Accepted Answer

When the two headlines truly performed the same (z < 1), accuracy was 53.1%, essentially chance on 49.6 to 56.6. It climbs monotonically from chance to 74.7% as the real gap widens, which is what makes the aggregate 64.5% result meaningful.

Question 6

Does Jev's stated confidence track its accuracy here?

Accepted Answer

Yes. Accuracy rises with stated confidence: 51.7% in the 0.50-0.60 band, 60.8% at 0.60-0.70, 67.9% at 0.70-0.80 and 78.5% at 0.80 and above. That lets you discard the bottom band and keep an edge worth having.

Question 7

How does Jev compare with common headline rules of thumb?

Accepted Answer

Jev scored 64.5% across 10,984 experiments, ahead of has-a-number at 58.5%, longer-headline at 54.2%, has-a-quotation at 51.9% and says-you-or-your at 50.3%. Shouted words won 41.6% of the time and a question mark only 35.5%.

Jev Headline Bench

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community