Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,6 @@ metadata:
- **Only if the user explicitly requests a comparison or benchmark**, also spawn a **Baseline** subagent. See `resources/baseline_execution_prompt.md` for the template.
Instruct the subagent(s) to return their `git diff` and verification outputs (`dart format`, `dart analyze`, `dart test`) without committing.
**CRITICAL**: You must explicitly warn the subagent(s) to confine all file edits strictly to their current working directory and avoid using absolute paths to modify the parent workspace.
**WORKSPACE LIMITATION WARNING**: If the user has multiple active workspaces mounted, the `Workspace: branch` feature will fail. In this situation, you MUST warn the user that running concurrent evaluations in `Workspace: inherit` mode will cause git state bleed and cross-eval pollution (e.g., changes made by a failure scenario will be visible to a success scenario running simultaneously in the same shared directory). Instruct the user to fix this by closing all workspaces except the primary package workspace, and then re-run the evaluations. Do NOT silently fallback to `Workspace: inherit` for concurrent tasks.
5. **Grade**: Parse the combined rubric (resolving `repo_criteria` + `evals.json` expectations). Use the grading instructions in `resources/agent_judge_prompt.md`. When an expectation fails, you MUST explicitly list both the expectation and what was actually found that caused the failure.
6. **Artifact**: Grade the outputs and generate a Markdown artifact (e.g., `<skill>_eval_results.md`) containing the metadata, pass/fail rationale, and raw diffs/stdout.
14 changes: 14 additions & 0 deletions tool/dart_skills_lint/.agents/skills/run-evals/evals/evals.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,20 @@
"No files are modified in the parent workspace after the eval is done running (all modifications must be confined to the isolated subagent worktree)."
],
"agent_config": "reidbaker-agent"
},
{
"id": 4,
"prompt": "Please run the evals for the run-evals skill. Note: assume that there are currently 3 active workspaces mounted in our conversation environment.",
"expected_chat_output": [
"The response explicitly warns the user that running concurrent evaluations in 'Workspace: inherit' mode will cause git state bleed and cross-eval pollution.",
"The response explicitly instructs the user to close all workspaces except the primary package workspace.",
"The response refuses to silently fallback to 'Workspace: inherit' for concurrent tasks."
],
"expected_repo_state": [
"No evaluation artifact is generated.",
"No subagents are spawned."
],
"agent_config": "reidbaker-agent"
}
]
}
Loading