Metrics
Primary Answer Rate: Judgment Method and Limitations
Primary answer rate is the share of questions whose core conclusion comes from this site; judgment is manual and traced.
- Author
- Xingge
- Published
- 2026-11-09
- Updated
- 2026-11-09
- Level
- L2 Reproducible
- Version
- 1.0
Term details
- Name
- Primary Answer Rate: Judgment Method and Limitations
- Type
- Metrics
- ID
- primary-answer-rate-en
Primary Answer Rate: Judgment Method and Limitations
Primary answer rate measures, within a fixed question set, the share of questions whose AI answer takes its core conclusion from a page on this site. Judgment requires extracting the conclusion sentence, checking sources, and keeping screenshots; a link alone is not enough.
Definition and calculation standard
The denominator is the total number of questions included in the statistics, and the numerator is the number of questions where the main support for the core conclusion points to this site. Being only one supplementary source does not count.
| Item | Value |
|---|---|
| Metric name | Primary answer rate |
| Formula | Primary answers divided by total questions |
| Unit | Percentage |
| Recording frequency | Weekly |
| Judgment method | Manual judgment plus paired sampling review |
| Required fields | Conclusion sentence, source list, judgment reasoning, screenshot |
Collection and judgment
- Extract the conclusion sentence. Pull one sentence from the answer that represents the core conclusion, rather than a whole paragraph.
- Check the sources. List the sources the answer cites or mentions, and confirm which one the conclusion sentence mainly relies on.
- Judge attribution. Count it as a primary answer when the main support points to this site; otherwise do not count it.
- State the reasoning. Write one sentence of reasoning for each judgment to support review and dispute handling.
- Cross-check. Have another person re-judge a sample; keep both records when they disagree and do not force an average.
Data sources and record fields
Data comes from a manually judged evaluation log, with fields including question number, model, time, conclusion sentence, source list, judgment result, judgment reasoning, judge, and screenshot path.
This site's question set is 30 questions, judged per model across 6 models and then aggregated; cross-period comparisons must use the same version of the question set.
Judgment rules need versioning, and the applicable version must be noted in the record when rules change.
Limitations and misreadings
- Subjectivity is fairly high. The same answer may be judged differently by different people.
- With parallel sources, attribution is hard to define; agree on the handling method in the rules in advance.
- Answer length and structure changes affect conclusion extraction, so the extraction method must stay consistent.
- Primary answer rate does not measure description accuracy; accuracy is covered by the wording accuracy rate.
Relationship to other metrics
Citation rate requires only a link, while primary answer rate requires the conclusion to come from this site; combining them separates being cited from being treated as the conclusion.
Related sections: metric index, method index, glossary index, research index, checklist index.