AstroAKD-Bench

Current documentation for generating astronomy-search benchmark questions, collecting observable archive evidence, and evaluating AstroAKD across three agent configurations.

Start with Getting started, including full installation of the external Astroquery MCP server. The independent Astroquery MCP server provides the archive tools; an ADS token must be configured on that server if you require literature evidence.

User guide

• Getting started
  • Prerequisites
  • Install the external Astroquery MCP server (Terminal 1)
  • Install and check AstroAKD-Bench (Terminal 2)
  • Recommended 100-question workflow
  • Current v2 reference coverage and literature drafts (2026-10-19)
  • What is measured?
  • Match-rate figure and selective MCP-only retry
  • New generation policy
• Create synthetic astronomical questions
  • Keep paid GPT batches after a failure
  • Long reference timeouts in a single command
  • Reference-aware synthetic questions (0.6.7.11.10)
  • Durable archive feasibility progress and fallback (0.6.7.11.10)
• ADS and MCP connectivity
  • Install and start the local MCP service
  • Check before generation
  • Required vs optional ADS
  • MCP endpoint
• Reference collection and scientific review
  • Completeness reports and plots
  • Review before scoring
  • Feasibility gate versus collected references
  • Independent multi-step evidence (0.6.7.11.11)
  • Multi-step evidence closure and literature draft report (0.6.7.11.12)
• Question-aware automatic reference-answer scoring (0.6.7.11.12)
• Run and report the A/B/C benchmark
  • Offline reports and figures
  • Multi-archive scoring closure in 0.6.7.11.12
• Python API

Development

• Build and publish the current documentation

What the results mean

Archive observations are unreviewed evidence, not automatically verified scientific answers. A/B/C completion and tool-use plots measure agent behavior; independently reviewed gold references are required to report scientific accuracy.

The README and CHANGELOG.md cover the current interface and release changes. Historical version-specific documentation is intentionally excluded from the HTML site and preserved separately as archival evidence.

Frozen prompts: instructions/astroakd/AstroAKD_Instruction.md (arm B) and instructions/astroakd_bench/AstroAKD-Bench_Instruction.md (generation only).