Every figure was reviewed three times: with the methods caption stripped, with the caption a journal would print, and with the prompt and source that produced it. The picture is identical in all three. Only the provenance changes.
d′ — discriminability. A hit rate is worthless without a criterion: our reviewers also call sound figures flawed 25% of the time once they see the code, so “recall went up” could just mean “it says flawed more often.” d′ folds the false alarms in. d′ = 0 is a coin flip.
Blind, a reviewer is below zero — they cannot discriminate at all. Hand them the methods caption, and they can: it moves d′ by +1.21 — more than the code adds on top of it. A truthful methods caption really does work.
We previously published the opposite (“the caption is worth nothing”). That was our bug, not a finding — our false-alarm judge was convicting models for correctly describing a clean figure. So why do you still need the code? Because a caption is a promise, and it can lie. Look at the next card.
The line is zero. Two of the three arms sit on the wrong side of it.
The line is zero. Looking harder does not get you off it. The caption does — until the caption is the thing that is lying to you.
Recall on the flaws we predicted were invisible, from figure alone to figure + code. Read this with the false-alarm rate below, never on its own.
Peer review gets the caption. It does not get the code. And the caption is an unverifiable promise: where it tells the truth it helps enormously, where it lies it is worth nothing, and where it launders the flaw it is worth less than nothing. You cannot tell which case you are in. Code is not a promise — it is a record. It cannot lie about what it did, because it is what it did.
The control. If provenance were simply making models sharper across the board, these would climb too. The question is whether they do.
Cosmetic flaws are in the pixels. You can see them by looking. That is exactly what separates them from every fatal flaw in this corpus.
circular-heatmap, blind → caption → code. It is the only level where provenance makes reviewers worse. Showing them the source that produced it lowered recall.
top50 <- head(order(res$padj), 50) pheatmap(z)
That is the circularity — select genes because they differ by group, then plot them to show they differ by group. It is also exactly what a normal RNA-seq paper does. The code looks routine because in this field it is routine.
Per-model, blind: haiku 0/3 · opus 3/3 · sonnet 2/3. With the caption: haiku 0/3 · opus 0/3 · sonnet 0/3. We do not yet know whether the caption suppresses suspicion or whether the stripped caption provokes it — the blind critiques are hedged (“the genes are almost certainly top-DE”), which is consistent with both. Stated as a hypothesis, not a result.
haiku’s false-alarm rate on genuinely sound figures, once it has the code. A model that misses the fatal set and invents flaws in the clean set is worse than useless: it trains you to ignore it.
A clean figure’s caption is how a reviewer confirms it is clean — you cannot see “BH-adjusted, blocked on donor” in a scatter of points. Take the caption away and the reviewer can only guess. Soundness is no more visible in a figure than a flaw is.
We labelled each flaw, in advance, as visible in the figure or not. The blind arm is that prediction’s only honest test. It failed on 1 of 10 levels, marked below. We were wrong; the rows stay.
| Level | Severity | We predicted | Figure alone | + caption | + code | Gap |
|---|---|---|---|---|---|---|
| nominal-p-as-adjusted | fatal | invisible | 0% | 0% | 100% | +100 |
| ora-length-bias | fatal | invisible | 0% | 22% | 0% | 0 |
| batch-confounded | fatal | invisible | 11% | 56% | 67% | +56 |
| dropped-blocking-factor | serious | invisible | 22% | 67% | 67% | +45 |
| roc-auc | fatal | invisible | 44% | 100% | 100% | +56 |
| circular-heatmap | fatal | invisible — wrong | 56% | 0% | 0% | -56 |
| bar-of-means | cosmetic | visible | 67% | 67% | 33% | -34 |
| truncated-axis | cosmetic | visible | 67% | 67% | 67% | 0 |
| pca-raw-counts | fatal | visible | 78% | 100% | 100% | +22 |
| km-optimal-cutpoint | fatal | visible | 89% | 100% | 100% | +11 |
Method: 378 graded verdicts, 3 models × 3 arms × 3 trials over 10 flawed levels and the clean controls. Figures are served to the reviewer from a scrubbed directory outside the repository under meaningless names — an earlier version of this eval leaked the answer in the image filename, and in the methods caption, and we only found out by reading the models’ own transcripts. Both are fixed. The numbers above are from the fixed run.