VIGIL — Offline Extractor Self-Improvement

VIGIL — Offline Extractor Self-Improvement An architecture diagram generated by Archify. Validation dev sample · Seeded 200 rows · not holdout · Architecture component Validation dev sample Seeded 200 rows · not holdout Score coded fields · Macro-F1 + confusion list · Architecture component · deterministic Score coded fields Macro-F1 + confusion list deterministic Evaluator Agent · Rewrites Extractor instruction · Architecture component · returns text only Evaluator Agent Rewrites Extractor instruction returns text only Candidate prompt · Saved under config/prompts/ · Architecture component · same dev sample Candidate prompt Saved under config/prompts/ same dev sample Guards pass? · F1 · coverage · anti-gaming · Architecture component · fail → discard Guards pass? F1 · coverage · anti-gaming fail → discard Dev gain? · Beats current prompt · Architecture component · no → discard Dev gain? Beats current prompt no → discard Locked holdout · Read after candidate fixed · Architecture component · improved? Locked holdout Read after candidate fixed improved? Promote · active.yaml v1 → v2 · Architecture component · Extractor only Promote active.yaml v1 → v2 Extractor only Discard · Keep candidate as evidence · Architecture component · no live change Discard Keep candidate as evidence no live change eval/runs/*.json · Every outcome recorded · Architecture component · promotion ledger eval/runs/*.json Every outcome recorded promotion ledger config/frozen.yaml · Risk thresholds · Architecture component · NEVER WRITTEN config/frozen.yaml Risk thresholds NEVER WRITTEN

Hard boundaries

  • • Only the Extractor prompt is revisable
  • • The locked holdout is read after the candidate is fixed
  • • config/frozen.yaml is outside the loop and never written

Measured result

  • • v1 dev macro-F1: 0.0056 → promoted v2: 0.4099
  • • Locked holdout macro-F1: 0.0081 → 0.4219
  • • Every promotion, discard, and guard block is written to eval/runs/

Offline only

  • • This loop never runs inside the Cloud Run triage path
  • • It cannot retune Analyst, Risk Agent, or Citation Critic
  • • The live safety threshold remains a fixed audit policy