Every AI research tool makes the same promise: it reads hundreds of papers for you. That claim cannot be checked. It is not false, but there is no way to falsify it either.
So we measured. Five Gemini models, spanning two generations, read the same 162 scientific papers and were scored on the same criteria set. The result contradicted our expectations in three separate ways.
This is the companion to Why “no fabricated sources” is a feature, not a slogan. That post cites other people’s research on models fabricating citations. This one is our own measurement of five models doing it.
The test
We used a dataset that already has answers: 162 paper × hypothesis decisions that a scientific community labelled and peer-reviewed among themselves. Each paper was asked the same question:
Does this paper test the hypothesis, and at what remove: directly, indirectly, or not at all? And which sentence in the paper shows that?
Three labels, heavily skewed: 3 papers testing a hypothesis directly, 121 indirectly, 38 not at all.
Each paper was fetched once and handed identically to all five models. Had each model searched for its own sources, one flaky retrieval would have looked like a weak model. The whole run used temperature 0, one prompt, one scorer: 36 minutes, 0 errors, 95 of 162 papers with open-access full text.
The criteria, and why each was chosen
The hard part is not running the measurement. It is choosing what to measure.
| Criterion | Why we chose it |
|---|---|
| macro-F1 | Weights all three labels equally instead of letting the common one dominate. Given the skew, this is the headline metric. |
| Relevant vs irrelevant | The decision that actually matters to a reader: keep the paper or discard it. It is also the only split with enough samples to conclude anything (124 against 38). |
| Real quotes | The share of quotes that genuinely exist in the source paper, checked mechanically. This is the anti-fabrication metric, and it does not depend on the model. |
| Evidence volume | Valid quotes extracted from the same 162 papers. Fewer is not wrong, but the reader receives noticeably less material to work with. |
| Accuracy | Published for completeness, but not used as a gate. The reason is immediately below. |
Why accuracy was disqualified as the headline
A model that reads nothing at all and answers “indirect” every single time scores 0.747 accuracy on this dataset, purely because 121 of 162 papers carry that label.
If a number makes your product look good and a model guessing blindly also achieves it, that number is not measuring your product.
The scoreboard
| Model | macro-F1 | Relevant vs irrelevant | Real quotes | Quotes | Cost per paper |
|---|---|---|---|---|---|
| Gemini 2.5 Flash | 0.468 | 0.885 | 0.884 | 626 | $0.0061 |
| Gemini 3.5 Flash | 0.480 | 0.858 | 0.857 | 180 | $0.0217 |
| Gemini 3.7 Flash | 0.468 | 0.885 | 0.924 | 257 | no published price |
| Gemini 2.5 Pro | 0.418 | 0.820 | 0.770 | 301 | $0.0223 |
| Gemini 3.1 Pro | 0.444 | 0.853 | 0.898 | 168 | $0.0344 |
| Guessing every time | 0.285 | n/a | n/a | 0 | $0 |
Three things that table says
1. Price does not buy quality
Gemini 3.1 Pro costs 5.6 times Gemini 2.5 Flash and loses on all three gate metrics.
The second most expensive model, Gemini 2.5 Pro, was the weakest in the whole set. It is also the only model whose accuracy, 0.710, sits below what blind guessing achieves at 0.747. If someone reports that this model scored 71%, that number is empty.
2. Newer is not better here, but it is more honest
Gemini 2.5 Flash, released in 2025, and Gemini 3.7 Flash, released this week, tie exactly on all three gate metrics: macro-F1 0.468, relevant-vs-irrelevant 0.885, accuracy 0.796.
The one meaningful difference is honesty. The newer model fabricates a third less: 7.6% against 11.6%.
That pattern is worth sitting with. If two generations of the same model family score identically on the task itself and differ mainly in caution, then capability on structured domain work may saturate earlier than the release cadence suggests, and what still separates models is disposition rather than reasoning.
3. There are cautious models and generous ones
Across the same 162 papers, the number of quotes extracted varies by a factor of 3.7: Gemini 2.5 Flash returned 626, Gemini 3.1 Pro returned 168.
No model is simply best. One hands you a lot of material to read yourself; another hands you a few sentences it is confident about. Knowing which you need is the reader’s job, not a leaderboard’s.
Which model fabricated most
Every quote is matched back against the source text. A changed digit, a flipped “increase” into “decrease”, or an invented sentence is discarded before it reaches a reader.
| Model | Quotes produced | Discarded as non-existent |
|---|---|---|
| Gemini 3.7 Flash | 278 | 7.6% |
| Gemini 3.1 Pro | 187 | 10.2% |
| Gemini 2.5 Flash | 708 | 11.6% |
| Gemini 3.5 Flash | 210 | 14.3% |
| Gemini 2.5 Pro | 391 | 23.0% |
Nearly a quarter of Gemini 2.5 Pro’s quotes did not exist in the paper it had just read.
What we have not achieved
A report that publishes only the flattering half is not a report. These three limits come from the same run, and they decide what the product is allowed to promise.
Rarely misses: 0.927. The share of genuinely relevant papers it keeps. This is the safe direction for a screening tool. Better to forward too much than to drop the paper you needed.
But over-includes: 0.421. The share of irrelevant papers it correctly rejects. You still discard roughly half of what it forwards. It reads first, you decide.
All five models are blind to one label: 0.000. No model, across two generations, has ever assigned the “tests it directly” label. That is systematic rather than one model’s quirk, and we do not yet know why. So we do not advertise that capability.
Why a cheap model is still safe
The anti-fabrication check does not live in the model. It is a text-matching pass that runs after every answer. Whichever model fabricates, that model’s fabrications are cut, including the most expensive one, including the one fabricating 23%.
That is why we can pick a cheap model, and also why we publish the unflattering numbers. An evidence tool that will not publish its own measurements is asking you to trust it rather than handing you evidence. If you want to run the same kind of scrutiny on your own dataset, see how to analyze research data.
Method note: the dataset is 162 paper × hypothesis decisions peer-reviewed and merged by the MecCog community, covering five hypotheses about APOE and neural mechanisms. Model names are the real API identifiers, not marketing names. Costs are converted from tokens actually spent. Gemini 3.7 Flash had no published price in our pricing table at the time of the run, so that cell is left blank rather than estimated. Raw per-paper results are stored alongside the source.