AstroAKD-Bench
Current documentation for generating astronomy-search benchmark questions, collecting observable archive evidence, and evaluating AstroAKD across three agent configurations.
Start with Getting started, including full installation of the external Astroquery MCP server. The independent Astroquery MCP server provides the archive tools; an ADS token must be configured on that server if you require literature evidence.
User guide
- Getting started
- Prerequisites
- Install the external Astroquery MCP server (Terminal 1)
- Install and check AstroAKD-Bench (Terminal 2)
- Recommended 100-question workflow
- Current v2 reference coverage and literature drafts (2026-10-19)
- What is measured?
- Match-rate figure and selective MCP-only retry
- New generation policy
- Create synthetic astronomical questions
- ADS and MCP connectivity
- Reference collection and scientific review
- Question-aware automatic reference-answer scoring (0.6.7.11.12)
- Run and report the A/B/C benchmark
- Published example:
astroakd_100_v2 - Python API
Development
Example benchmark results
The published astroakd_100_v2 example contains all 100 questions, 92 independently retrieved reference-answer sets, per-question A/B/C classifications, and the generated figures below.

The percentages measure automatic agreement with saved archive evidence, not independently verified scientific accuracy. Eight literature tasks remain outside the scoring denominator.
What the results mean
Archive observations are unreviewed evidence, not automatically verified scientific answers. A/B/C completion and tool-use plots measure agent behavior; independently reviewed gold references are required to report scientific accuracy.
The README and CHANGELOG.md cover the current interface and release changes. Historical version-specific documentation is intentionally excluded from the HTML site and preserved separately as archival evidence.
Frozen prompts: instructions/astroakd/AstroAKD_Instruction.md (arm B) and instructions/astroakd_bench/AstroAKD-Bench_Instruction.md (generation only).