Skip to content

Goal Source Rationales Runbook

Status: PoC-to-runtime runbook

Scope: generated human-readable source rationales for selected, all-relevant, and public source-backed SkillPilot learning goals

Purpose

This runbook describes how to run and inspect the first source-rationale PoC.

The generator answers this review question for one or more learning goals:

How can a human reach the original curriculum source, inspect the extracted source evidence, and understand why the SkillPilot goal is justified by that evidence?

The current PoC deliberately uses only the classic reviewed SkillPilot route:

canonical goal -> mapping review -> source extraction -> official source document

MEM/FWU SPARQL is rendered as an explicit unavailable route until the classic route is stable enough to compare against.

Default Command

Run the canonical Mathematik PoC from the app directory:

npm run quality:goal-source-rationales:poc

This writes:

  • docs/qa-ci/status/goal-source-rationales-poc.md
  • docs/qa-ci/status/goal-source-rationales-poc.json

The default scope is canonical Gymnasium Mathematik with jurisdiction DE-BY and one selected PoC goal.

Custom Goal Set

The script can render one or more selected goal IDs:

npx tsx scripts/generateGoalSourceRationales.ts \
  --jurisdiction=DE-BY \
  --goal=<goal-id> \
  --goal=<another-goal-id>

or:

npx tsx scripts/generateGoalSourceRationales.ts \
  --jurisdiction=DE-BY \
  --goals=<goal-id>,<another-goal-id>

Useful options:

  • --landscape=<path> selects a different landscape JSON.
  • --mapping-root=<path> selects the mapping-review root to scan.
  • --output-json=<path> changes the JSON output.
  • --output-md=<path> changes the Markdown output.
  • --no-md skips Markdown output for runtime-only JSON indexes.
  • --goals=source-backed derives all target goals that have reviewed source mappings in the selected scope.
  • --goals=source-backed-relevant-leaves derives all relevant Mathematik leaf goals with at least one reviewed classic source mapping; --jurisdiction is then used as preferred rendered route, not as the item-set filter.
  • --goals=all-relevant-leaves derives all relevant Mathematik leaf goals and renders missing classic source routes as gap entries.
  • --jurisdiction=all disables jurisdiction filtering.
  • --include-mem adds a live MEM/FWU-SPARQL comparison where configured.
  • --mem-config=<path> selects the MEM/FWU consistency config.

MEM/FWU Showcase

Run the current Mathematik showcase with live MEM/FWU-SPARQL matches:

npm run quality:goal-source-rationales:mem-examples

This writes:

  • docs/qa-ci/status/goal-source-rationales-mem-examples.md
  • docs/qa-ci/status/goal-source-rationales-mem-examples.json

The showcase currently uses the Bayern Mathematik/Gymnasium LehrplanPLUS comparison because that scope is aligned in the MEM consistency audit. The generated rationale remains conservative: MEM/FWU is shown as a consistent alternate route, while the classic reviewed source route remains the primary provenance path.

For a non-technical reader-facing variant, run:

npm run quality:goal-source-rationales:mem-examples:plain

This writes:

  • docs/qa-ci/status/goal-source-rationales-mem-examples-plain.md
  • docs/qa-ci/status/goal-source-rationales-mem-examples-plain.json

This variant hides SkillPilot goal IDs in the visible explanation, names the learning goal by its SkillPilot path, and includes a concrete SPARQL query plus result-reading instructions for every MEM/FWU match.

Runtime Public Index

The Explorer source-rationale button does not call MEM/FWU or the local QA dashboard at render time. The production UI loads a Vite-built JSON asset from app/src/data first, so the index is shipped through the same /assets/... path family as the application code. A copy is also written to /data/... for local direct checks and as a legacy fallback:

/assets/goal-source-rationales-math-public-<hash>.json
/data/goal-source-rationales-math-public.json

Regenerate the current public Mathematik index with:

npm run quality:goal-source-rationales:math-public

Verify that the runtime copies are present, identical, cover the currently source-backed relevant leaf-goal set, and still keep the MEM/FWU showcase goals consistent with:

npm run check:goal-source-rationales:math-public

After npm run build, verify that Vite emitted the hashed JSON asset and that the built JavaScript references it:

npm run check:goal-source-rationales:build-artifact

After deployment, verify the live host in the same shape the browser uses:

npm run smoke:goal-source-rationales:deployment

For another host, pass a base URL:

npm run smoke:goal-source-rationales:deployment -- --base-url=http://127.0.0.1:8080

The repository deploy script runs this smoke test after restarting the production service. Override the target host with:

SKILLPILOT_BASE_URL=https://staging.example.org scripts/deploy.sh

This command currently emits the gap-free public runtime subset:

relevant canonical Mathematik leaf goals with at least one reviewed classic source mapping

The rendered route still prefers DE-BY where available so that the configured MEM/FWU Bayern comparison remains visible. If no Bayern route exists for a goal, the generator falls back to another reviewed classic source route.

It writes:

  • app/src/data/goal-source-rationales-math-public.json
  • app/public/data/goal-source-rationales-math-public.json

The current index is a scalable runtime step, not the final national coverage promise. It expands the UI from three hand-picked PoC goals to all currently source-backed relevant Mathematik leaf goals that the generator can derive from mapping reviews. Remaining uncovered leaf goals stay in the all-relevant report and gap-issue queue; relation rationales for requires and contains are still a separate rollout lane.

Review Checklist

For every rendered goal, inspect:

  • sourceRationaleStatus
  • classic_source_reviewed is acceptable for the current public runtime index.
  • classic_source_partial is usable only if the mapping rationale explains the partial shape.
  • classic_source_gap means a non-blocking follow-up issue is needed.
  • Originalquelle Finden
  • official URL or retained source path is present
  • source reference or searchable source hint is concrete enough
  • Extrahierter Quellenbeleg
  • source document metadata is resolved
  • source goal ID and source reference match the mapping decision
  • Markdown excerpt is short; full extracted text remains in JSON
  • Warum das SkillPilot-Ziel begründet ist
  • mapping rationale is present and understandable
  • exact, partial, split, or aggregate mappings are not overstated
  • MEM/FWU SPARQL-Route
  • mem_sparql_unavailable is expected for the classic-only PoC
  • mem_sparql_consistent is expected for the configured Bayern showcase examples
  • do not treat MEM/FWU as primary evidence until a concrete scope has been accepted as primary-ready

Follow-Up Issues

Create a manual review issue when:

  • no classic source route is found for a requested visible atomic goal
  • the route is only partial and the rationale does not explain the mapping shape
  • the official source URL or retained source path is missing
  • source document metadata cannot be resolved
  • a MEM/FWU route later exists but disagrees with the classic route

These issues are non-blocking during the PoC. They become candidates for blocking QA only after SkillPilot defines a maturity level that promises public source explainability for visible learning goals.