AstroAKD-Bench

Current documentation for generating astronomy-search benchmark questions, collecting observable archive evidence, and evaluating AstroAKD across three agent configurations.

Start with Getting started, including full installation of the external Astroquery MCP server. The independent Astroquery MCP server provides the archive tools; an ADS token must be configured on that server if you require literature evidence.

Example benchmark results

The published astroakd_100_v2 example contains all 100 questions, 92 independently retrieved reference-answer sets, per-question A/B/C classifications, and the generated figures below.

Strict archive-reference agreement of the three agents for 92 evaluable questions

The percentages measure automatic agreement with saved archive evidence, not independently verified scientific accuracy. Eight literature tasks remain outside the scoring denominator.

What the results mean

Archive observations are unreviewed evidence, not automatically verified scientific answers. A/B/C completion and tool-use plots measure agent behavior; independently reviewed gold references are required to report scientific accuracy.

The README and CHANGELOG.md cover the current interface and release changes. Historical version-specific documentation is intentionally excluded from the HTML site and preserved separately as archival evidence.

Frozen prompts: instructions/astroakd/AstroAKD_Instruction.md (arm B) and instructions/astroakd_bench/AstroAKD-Bench_Instruction.md (generation only).