Skip to content

Groundedness benchmark

Measured and reproducible. Not a self-reported score.

Many AI tools self-publish high benchmark scores nobody can verify, and are losing trust for it. Phở does the opposite: each number below carries a run date, a source commit, and a command to reproduce it yourself. If a number is stale, you can see it.

Measuring — no number published yet.

This page only shows a number from a real run, with its date and commit. No public run has happened yet, so no number is shown (we do not put a placeholder figure). The question set is ready: 56 Vietnamese medical/research questions.

What this number means (and does not)

Groundedness = supported / verifiable, where verifiable = supported + partial + unsupported (unverifiable claims are dropped from the denominator, fail-safe). It measures whether each claim in the answer is actually backed by the abstract it cited. In short: the not-fabricated rate.

It is not a "percent medically correct" score. An answer can be grounded (faithful to the source it cites) while that source still has limits. This measures faithfulness to sources, exactly where AI tools tend to fail.

How we measure it

  1. 56 Vietnamese medical/research questions (the vn-grounding + vn-academic-extended (VN medical + research questions) set).
  2. Translate to English, search PubMed + OpenAlex (keyless public APIs).
  3. Answer grounded on the retrieved abstracts, each claim carrying a citation.
  4. A separate LLM judging pass (same model, seeing only the claim + the abstract it cited) scores each claim: supported / partial / unsupported.

Because generation and judging use an LLM, the number varies slightly run to run, so it is a point-in-time snapshot (with date + commit), not a constant.

Reproduce it

Needs GOOGLE_GENERATIVE_AI_API_KEY and egress to PubMed/OpenAlex/Gemini:

GOOGLE_GENERATIVE_AI_API_KEY=… pnpm --filter @pho/agent-eval eval:grounding:academic:publish

Method + metric definition: eval docs.

Nội dung minh hoạ năng lực công cụ, không phải tư vấn điều trị cho ca bệnh cụ thể. Bệnh nhân trong case study là dựng lại (synthetic). Quyết định lâm sàng thuộc về bác sĩ.