Clinical Research Bench
Do frontier models cite real evidence for medical questions? We checked every citation.
Launch edition · 2026-09-23 · 5 models · 15 Cochrane questions + 5 invented-drug traps · one run. A snapshot: read the counts, not only the percentages.
best to worst model. Invalid = the cited PMID/DOI does not exist, or exists but is about something else.
down from 16 when asked directly. An overclaim is a firm conclusion where the 2026 Cochrane review says the evidence is insufficient.
every model said the invented drug, trial or biomarker was unknown, in both rounds.
Round 1: ask the model directly
Each model answered the same 15 questions, told to cite PMIDs or DOIs. No tools. We resolved every ID in PubMed and the DOI registry, then checked whether the paper is about the question at all.
| Model | Invalid citations | of which real but off-topic | Claims supported by their source | Overclaimed / contradicted |
|---|---|---|---|---|
| Claude Opus 5.5 | 3% (1/37) | 0 | 37% (29/79) | 4 / 1 |
| GPT-6 Sol | 17% (5/29) | 4 | 23% (15/66) | 1 / 1 |
| Grok 4.7 | 22% (5/23) | 4 | 12% (5/41) | 4 / 2 |
| GPT-6 Luna | 42% (13/31) | 10 | 10% (9/90) | 1 / 1 |
| Gemini 3.8 Flash | 91% (59/65) | 57 | <1% (1/209) | 6 / 0 |
Round 2: the same model through Phở
Phở retrieves real papers first, then the model answers only from them and cites them. Every model got the identical evidence (verified on 20/20 questions), and the Cochrane review used as the answer key, with all its earlier versions, was removed from what Phở could find.
| Model | Supported claims: direct → through Phở | Overclaimed | Contradicted |
|---|---|---|---|
| Claude Opus 5.5 | 37% → 68% (167/246) | 4 → 0 | 1 → 0 |
| GPT-6 Sol | 23% → 35% (19/54) | 1 → 0 | 1 → 1 |
| Grok 4.7 | 12% → 55% (32/58) | 4 → 0 | 2 → 1 |
| GPT-6 Luna | 10% → 27% (13/48) | 1 → 0 | 1 → 1 |
| Gemini 3.8 Flash | <1% → 43% (19/44) | 6 → 0 | 0 → 0 |
How it works
- Questions from outside Phở. 15 Cochrane systematic reviews published in 2026, picked by a fixed random seed. The reference answer is the review authors’ own conclusion. 5 trap questions name a drug, trial or biomarker that has zero hits in PubMed, ClinicalTrials.gov and Crossref.
- Mechanical checks first. A citation is invalid if its ID does not resolve, or if the paper shares no domain term with the question (e.g. a laser-physics paper cited for a dental-implant question). No judge involved.
- One neutral judge for the rest. Jev, an evaluation model not made by any contestant, scores whether each cited abstract supports its sentence. On 200 held-out expert-labelled PubMedQA items it agreed with the experts 83% of the time (κ 0.67); a single medical annotator scored 78% on the same task.
- Pre-registered and reproducible. Questions, prompts, judge and scoring were frozen in commit
a50de2c9before the run. Transcripts, every cited ID and every verdict are public. Cost of the whole run: $2.13.
What this does not show
- Clinical correctness or safety. "Supported" means the cited abstract backs the sentence, not that the medicine is right. No clinician reviewed the question set.
- A precise ranking. 15 questions, one run: differences of a few counts are ties.
- Round 2 numbers look modest because the answer key was deliberately removed. Many through-Phở answers correctly said the retrieved studies did not settle the question.
Data, transcripts and method
This benchmark measures whether cited sources support what a model wrote. It is not medical advice and not a clinical safety evaluation. Clinical decisions belong to clinicians.