Case file · track record
How well does the detector work?
Results of the classifier on a frozen, held-out test set that was never used for prompt tuning. Every number shows its sample size n and a 95% Wilson interval (drawn as an error bar). Each data source is reported separately; they are never merged into one number.
Loading results...
This page only shows numbers from a real run of the classifier on the frozen test set. Until that run exists, there is nothing to show, so no charts are drawn.
How to read this page
- False-positive rate (FPR) on "no technique": of the texts that the source's annotators labelled as ordinary information (human annotators for SemEval, a single LLM annotator for the real-headline source), the share the system flagged with some technique. Lower is better. This is the number that tells you how often an honest headline gets wrongly accused.
- Precision of a technique: of the texts the system labelled with it, the share that really have it. Recall: of the texts that really have it, the share the system found.
- Wilson interval: the 95% range where the true value plausibly lies given n. Small n gives wide intervals: do not over-read differences whose intervals overlap.
- Confidence (High, Moderate, Disputed) is how strongly the system agrees with itself; it is not a probability. The calibration table checks whether "High" really is more accurate than "Moderate". See Methodology below.
- Disputed means the system could not settle on an answer; such texts are never shown as a confident verdict.
Human vs model
After each clipping, players can dispute the reference answer. A dispute is one player's opinion, not a label: it changes no reference answer and none of the numbers above. Disputes that survive a team review are added to the development set only, never to the frozen test set. Counts only: comments and player identities are never shown.
Loading dispute counts...
* interval computed in the browser from k and n (Wilson, 95%). Disputes per answer are not independent trials, so read the interval as a rough guide.
Known limitations
- Domain shift: public datasets label sentences from full news articles or memes, while the game uses headlines.
- Public datasets may have leaked into the training data of the models, which can inflate their scores.
- Techniques outside our six (for example flag-waving, slogans, repetition) are excluded from evaluation rather than counted as "no technique".
- Small samples: intervals are wide, especially per technique.
Methodology
The classifier is an ensemble of three different large language models from different vendors. Each model labels the text independently with the same prompt; a text is accused of a technique only when all three models name it. When only two of three see a technique, the text is not accused and is shown as Disputed, with the near-miss explained. Two of three saying "no technique" is a clean verdict. Agreement is not a probability, and models from the same era can share the same blind spots, so agreement can be confidently wrong.
Flagging policies: by default a text is flagged only when all three models name a technique (precision first: a false accusation costs more than a miss). The table also shows the looser two-of-three rule, computed from the same votes, so the trade-off is visible.