--- name: skill-creator description: Use when creating a skill from scratch, editing or improving an existing skill, verifying a skill works before deployment, running evals against a skill, benchmarking skill performance with variance analysis, or optimizing a skill's description for better triggering accuracy. --- # Skill Creator Audience: AI coding agents first. Optimize every token for model activation, not human aesthetics. Do not prettify this back into prose. Build and improve skills via RED → GREEN → REFACTOR. Same cycle as `tdd`. **REQUIRED BACKGROUND:** Understand `tdd` before using this skill. ## The Iron Law ``` NO SKILL WITHOUT A FAILING TEST FIRST ``` Applies to NEW skills AND edits to existing skills. Wrote skill before testing? Delete it. Start over. **No exceptions:** - Not for "simple additions" - Not for "just adding a section" - Not for "documentation updates" - Don't keep untested changes as "reference" - Don't "adapt" while running tests - Delete means delete **Violating the letter of the rules is violating the spirit of the rules.** ## Process 1. Decide scope + approach 2. RED — run pressure scenarios WITHOUT the skill, capture baseline failures verbatim 3. GREEN — write minimal skill addressing those exact failures 4. Test: create test prompts, run with-skill AND baseline 5. Evaluate: qualitative + quantitative 6. REFACTOR — close loopholes new testing exposes, re-verify 7. Repeat until satisfied; expand test set, retry at scale 8. Run description optimizer for triggering accuracy Read where the user already is → jump in there. "I want a skill for X" → narrow intent, baseline, draft, test. Draft exists → go straight to eval/iterate. **User communication:** read context cues. "evaluation"/"benchmark" are borderline OK; explain "JSON"/"assertion" unless the user clearly knows them. --- ## What is a Skill? **Skill = reference guide** for proven techniques, patterns, tools. **NEVER:** narrative about how you solved something once. **Create when:** technique wasn't intuitively obvious; reusable across projects; pattern applies broadly. **Don't create for:** one-off solutions; standard practices documented elsewhere; project-specific conventions (→ CLAUDE.md); mechanically enforceable constraints (→ automate instead). **Types:** Technique (concrete method with steps) · Pattern (way of thinking about problems) · Reference (API docs, syntax guides). ## TDD Mapping | TDD Concept | Skill Creation | |---|---| | Test case | Pressure scenario with subagent | | Production code | SKILL.md | | RED | Agent violates rule without skill (baseline) | | GREEN | Agent complies with skill present | | Refactor | Close loopholes while maintaining compliance | | Write test first | Run baseline scenario BEFORE writing skill | | Watch it fail | Document exact rationalizations agent uses | | Minimal code | Write skill addressing those specific violations | | Watch it pass | Verify agent now complies | | Refactor cycle | Find new rationalizations → plug → re-verify | --- ## Creating a Skill ### Capture Intent Understand intent first. Conversation already contains the workflow? Extract from history: tools used, step sequence, corrections, input/output formats. 1. What should the skill enable Claude to do? 2. When should it trigger? (phrases/contexts) 3. Expected output format? 4. Set up test cases? (objective outputs → yes; subjective → decide together) Ask about edge cases, I/O formats, example files, success criteria, dependencies. Check available MCPs — research in parallel via subagents. Come prepared; reduce burden on the user. ### Skill Anatomy ``` skill-name/ ├── SKILL.md (required) │ ├── YAML frontmatter (name, description required) │ └── Markdown instructions └── Bundled Resources (optional) ├── scripts/ - Executable code for deterministic/repetitive tasks ├── references/ - Docs loaded into context as needed └── assets/ - Files used in output (templates, icons, fonts) ``` **Progressive disclosure — 3 loading levels:** 1. Metadata (name + description) — always in context (~100 words) 2. SKILL.md body — in context when triggered (<500 lines ideal) 3. Bundled resources — as needed (unlimited; scripts execute without loading) Keep SKILL.md under 500 lines. Approaching the limit → add hierarchy with clear pointers. Reference files >300 lines → include a table of contents. **Domain organization** for multiple frameworks — Claude reads only the relevant reference: ``` cloud-deploy/ ├── SKILL.md (workflow + selection) └── references/{aws,gcp,azure}.md ``` **File organization by weight:** - All content fits inline → `SKILL.md` alone - Reusable tool → `SKILL.md` + `example.ts` (working helpers to adapt) - Heavy reference → `SKILL.md` + large `.md` refs + `scripts/` ### Frontmatter - `name`: letters, numbers, hyphens only. No parentheses or special chars. Must be UNIQUE across all skill directories — two skills declaring the same `name` collide and one is silently discarded. - `description`: max 1024 chars. Third person. Starts with "Use when…". - **CRITICAL: description = triggering conditions ONLY. NEVER summarize workflow.** Workflow-in-description creates a shortcut Claude takes instead of reading the body. - Make it pushy — include adjacent terms the user might say without naming the skill. - All "when to use" lives here, not in the body. ```yaml # DO NOT — summarizes workflow; Claude follows description, skips body description: Use when executing plans — dispatches subagent per task with code review between tasks # DO NOT — too abstract description: For async testing # DO NOT — first person description: I can help you with async tests when they're flaky # GOOD — triggering conditions only description: Use when executing implementation plans with independent tasks in the current session # GOOD — problem-focused, no workflow description: Use when tests have race conditions, timing dependencies, or pass/fail inconsistently # GOOD — tech-specific with explicit trigger description: Use when using React Router and handling authentication redirects ``` ### Body Template ```markdown --- name: skill-name-with-hyphens description: Use when [specific triggering conditions and symptoms] --- # Skill Name ## Overview Core principle in 1-2 sentences. ## When to Use [Small inline flowchart IF decision non-obvious] Symptoms and use cases. When NOT to use. ## Core Pattern Before/after comparison. ## Quick Reference Table or bullets for scanning. ## Implementation Inline code for simple patterns. Link to file for heavy reference. ## Common Mistakes What goes wrong + fixes. ``` ### Writing Patterns Use imperative form. Pin exact output shapes: ```markdown ## Report structure ALWAYS use this exact template: # [Title] ## Executive summary ## Key findings ## Recommendations ``` Show input→output pairs: ```markdown ## Commit message format **Example 1:** Input: Added user authentication with JWT tokens Output: feat(auth): implement JWT-based authentication ``` Explain WHY behind each instruction — models go beyond rote instructions when they understand reasoning. Yellow flag: ALL-CAPS ALWAYS/NEVER plus rigid structure → reframe with reasoning. Write draft, look fresh, improve. ### Claude Search Optimization (CSO) **Keyword coverage** — use words Claude would search for: error messages ("Hook timed out", "ENOTEMPTY", "race condition"), symptoms ("flaky", "hanging", "zombie", "pollution"), synonyms ("timeout/hang/freeze", "cleanup/teardown/afterEach"), actual commands, library names, file types. **Naming** — active voice, verb-first. `creating-skills` not `skill-creation`. `condition-based-waiting` not `async-test-helpers`. Gerunds work well for processes. **Token targets** — getting-started workflows <150 words; frequently-loaded <200 words total; other skills <500 words. Move flag details to `--help`, cross-reference, compress examples, cut redundancy. **Cross-referencing:** ```markdown # GOOD **REQUIRED SUB-SKILL:** Use tdd **REQUIRED BACKGROUND:** You MUST understand debug # DO NOT See skills/tdd # unclear if required @skills/tdd/SKILL.md # force-loads, burns context ``` **Discovery flow** — optimize for it; put searchable terms early and often: 1. Encounters problem ("tests are flaky") → 2. Finds SKILL (description matches) → 3. Scans overview → 4. Reads quick reference → 5. Loads example only when implementing. ### Flowcharts **ONLY for:** non-obvious decision points, process loops where you might stop early, "A vs B" decisions. **NEVER for:** reference material (→ tables), code examples (→ code blocks), linear instructions (→ numbered lists), labels without semantic meaning. Style conventions: `references/graphviz-conventions.dot`. Render: `node references/render-graphs.js `. ### Code Examples One excellent example beats many mediocre ones. Complete, runnable, from a real scenario, ready to adapt. Language by domain: testing → TypeScript/JS; system debugging → Shell/Python; data processing → Python. **DO NOT:** implement in 5+ languages; create fill-in-the-blank templates; write contrived examples. ### Anti-Patterns **DO NOT — narrative example:** "In session 2025-10-03, we found…" → too specific, not reusable. **DO NOT — multi-language dilution:** `example-js.js` + `example-py.py` → mediocre quality, maintenance burden. **DO NOT — code in flowcharts:** can't copy-paste, hard to read. **DO NOT — generic labels:** `helper1`, `step3` → labels MUST carry semantic meaning. --- ## Testing **Complete methodology:** `references/testing-skills-with-subagents.md`. Load it when setting up pressure scenarios or baseline runs. Test by skill type: | Type | Test with | Success criteria | |---|---|---| | Discipline-enforcing | Pressure scenarios combining time + sunk cost + exhaustion | Agent follows rule under maximum pressure | | Technique | Application, variation, missing-information scenarios | Agent applies technique to a new scenario | | Pattern | Recognition, application, counter-example scenarios | Agent identifies when/how to apply | | Reference | Retrieval, application, gap scenarios | Agent finds and correctly applies the information | ### Rationalizations for Skipping Testing | Excuse | Reality | |---|---| | "Skill is obviously clear" | Clear to you ≠ clear to other agents. Test it. | | "It's just a reference" | References have gaps. Test retrieval. | | "Testing is overkill" | Untested skills have issues. Always. 15 min testing saves hours. | | "I'll test if problems emerge" | Problems = agents can't use the skill. Test BEFORE deploying. | | "Too tedious to test" | Testing < debugging a bad skill in production. | | "I'm confident it's good" | Overconfidence guarantees issues. Test anyway. | | "Academic review is enough" | Reading ≠ using. Test application scenarios. | | "No time to test" | Deploying untested wastes more time later. | All of these mean: test before deploying. No exceptions. ### Test Cases After the draft, write 2-3 realistic test prompts. Share: "Here are test cases I'd like to try. Look right?" Then run. Save to `evals/evals.json`: ```json { "skill_name": "example-skill", "evals": [ {"id": 1, "prompt": "User's task prompt", "expected_output": "Description of expected result", "files": []} ] } ``` Full schema (including `assertions`): `references/schemas.md`. --- ## Running and Evaluating Test Cases One continuous sequence — do NOT stop partway. Do NOT use `/skill-test` or any other testing skill. **Bundled harness required.** Steps below call `scripts/`, `agents/`, `assets/`, and `eval-viewer/` from the upstream skill-creator package. They are NOT present in this local copy — fetch the upstream bundle into this directory before running the harness, or fall back to the manual review loop in "Claude.ai-Specific". Results go in `-workspace/`, sibling to the skill directory. Organize by iteration (`iteration-1/`), each test case in its own directory (`eval-0/`). Create as you go, not upfront. ### Step 1 — Spawn All Runs (with-skill AND baseline) in the Same Turn For each test case spawn two subagents in the same turn. NEVER spawn with-skill first then return for baselines. **With-skill run:** ``` Execute this task: - Skill path: - Task: - Input files: - Save outputs to: /iteration-/eval-/with_skill/outputs/ - Outputs to save: ``` **Baseline run** (same prompt): - New skill → no skill at all → `without_skill/outputs/` - Improving existing → old version (snapshot first: `cp -r /skill-snapshot/`) → `old_skill/outputs/` Write `eval_metadata.json` per test case: ```json {"eval_id": 0, "eval_name": "descriptive-name-here", "prompt": "The user's task prompt", "assertions": []} ``` ### Step 2 — While Runs Are In Progress, Draft Assertions Objectively verifiable, descriptive names. Subjective skills → skip assertions, use human judgment. Update `eval_metadata.json` and `evals/evals.json`. Explain to the user what they'll see. ### Step 3 — As Runs Complete, Capture Timing Save immediately to `timing.json` in the run directory: ```json {"total_tokens": 84852, "duration_ms": 23332, "total_duration_seconds": 23.3} ``` Only opportunity to capture — it arrives via task notification and is persisted nowhere else. ### Step 4 — Grade, Aggregate, Launch Viewer 1. **Grade** — spawn a grader subagent (reads `agents/grader.md`). Save `grading.json` per run. Fields MUST be `text`, `passed`, `evidence` — NOT `name`/`met`/`details`; the viewer depends on exact field names. 2. **Aggregate:** ```bash python -m scripts.aggregate_benchmark /iteration-N --skill-name ``` Produces `benchmark.json` + `benchmark.md`. Put with_skill before its baseline counterpart. 3. **Analyst pass** — read `agents/analyzer.md`. 4. **Launch viewer:** ```bash nohup python /eval-viewer/generate_review.py \ /iteration-N \ --skill-name "my-skill" \ --benchmark /iteration-N/benchmark.json \ > /dev/null 2>&1 & VIEWER_PID=$! ``` Iteration 2+: add `--previous-workspace /iteration-`. No display/headless: `--static ` for standalone HTML; feedback downloads as `feedback.json` on "Submit All Reviews". 5. **Tell the user:** "Opened results in browser. 'Outputs' tab: click test cases, leave feedback. 'Benchmark' tab: quantitative comparison. Come back when done." ### Step 5 — Read the Feedback ```json {"reviews": [{"run_id": "eval-0-with_skill", "feedback": "chart missing axis labels", "timestamp": "..."}], "status": "complete"} ``` Empty feedback = the user thought it was fine. Focus improvements on specific complaints. Kill the viewer when done: `kill $VIEWER_PID 2>/dev/null`. --- ## Improving the Skill 1. **Generalize** — the skill runs millions of times across many prompts. Avoid overfitting to test cases. Stubborn issue → try different metaphors or working patterns. 2. **Keep lean** — remove what isn't pulling weight. Read transcripts, not just final outputs. 3. **Explain why** — transmit understanding of what the user actually needs. ALL-CAPS ALWAYS/NEVER → yellow flag → reframe with reasoning. 4. **Bundle repeated work** — all 3 test cases wrote `create_docx.py`? Bundle it as `scripts/create_docx.py`. ### Closing Loopholes (REFACTOR) Agent found a new rationalization → add an explicit counter → re-test until bulletproof. ```markdown # DO NOT — leaves a loophole Write code before test? Delete it. # GOOD — closes all escapes Write code before test? Delete it. Start over. **No exceptions:** - Don't keep it as "reference" - Don't "adapt" it while writing tests - Don't look at it - Delete means delete ``` Build the rationalization table from baseline testing. Add a red-flags list. Update CSO with violation symptoms. Behavioral levers: `references/persuasion-principles.md`. ### Iteration Loop 1. Apply improvements 2. Rerun all test cases into `iteration-/`, baselines included 3. Launch reviewer with `--previous-workspace` pointing at the previous iteration 4. Wait for the user to review and say done 5. Read new feedback, improve, repeat Stop when: the user says they're happy, all feedback is empty, or there's no meaningful progress. ### Advanced: Blind Comparison Optional; the human review loop is usually sufficient. Read `agents/comparator.md` + `agents/analyzer.md`. Give two outputs to an independent agent without labels → judge quality → analyze why the winner won. --- ## Description Optimization After creating or improving a skill, offer to optimize its description for triggering accuracy. ### Step 1 — Generate Trigger Eval Queries Create 20 queries, mixed: ```json [{"query": "the user prompt", "should_trigger": true}, {"query": "another prompt", "should_trigger": false}] ``` Queries MUST be realistic — concrete and specific, with file paths, personal context, column names, company names, URLs. Mix lowercase/abbreviations/typos/casual. Mix lengths. Focus on edge cases. - **Should-trigger (8-10):** different phrasings of the same intent; cases where the user doesn't name the skill but clearly needs it. - **Should-not-trigger (8-10):** most valuable are near-misses that share keywords but need a different skill. Avoid obviously irrelevant negatives. User reviews and signs off. ### Step 2 — Review with User 1. Read the template from `assets/eval_review.html` 2. Replace `__EVAL_DATA_PLACEHOLDER__` → JSON array, plus `__SKILL_NAME_PLACEHOLDER__` and `__SKILL_DESCRIPTION_PLACEHOLDER__` 3. Write to `/tmp/eval_review_.html`, open it 4. User edits queries and exports → check `~/Downloads/eval_set.json` ### Step 3 — Run the Optimization Loop ```bash python -m scripts.run_loop \ --eval-set \ --skill-path \ --model \ --max-iterations 5 \ --verbose ``` Use the model ID from the system prompt — the triggering test MUST match what the user actually experiences. Periodically tail output for user updates. Loop: 60% train / 40% held-out, evaluates each query 3× for a reliable trigger rate, proposes improvements, re-evaluates, returns JSON with `best_description`. **How triggering works:** Claude sees the `available_skills` list (name + description) and decides whether to consult based on the description. Complex, multi-step, specialized queries trigger reliably when the description matches. Simple one-step queries may not trigger even with a matching description — so eval queries MUST be substantive. ### Step 4 — Apply Take `best_description` from the JSON output, update SKILL.md frontmatter, show before/after, report scores. --- ## STOP: Before Moving to the Next Skill After ANY skill, complete deployment before starting the next: - NEVER create multiple skills in a batch without testing each - NEVER move to the next skill before the current one is verified Deploying untested skills = deploying untested code. ## Checklist **IMPORTANT: use TodoWrite for EACH item.** **RED — write the failing test:** - [ ] Create pressure scenarios (3+ combined pressures for discipline skills) - [ ] Run scenarios WITHOUT the skill — document baseline behavior verbatim - [ ] Identify patterns in rationalizations/failures **GREEN — write the minimal skill:** - [ ] Name uses only letters, numbers, hyphens; unique across all skill dirs - [ ] YAML frontmatter with `name` and `description` (max 1024 chars) - [ ] Description starts with "Use when…" — triggering conditions only, NO workflow summary - [ ] Description in third person - [ ] Keywords throughout for search (errors, symptoms, tools) - [ ] Clear overview with core principle - [ ] Addresses the specific baseline failures from RED - [ ] Code inline OR linked to a separate file - [ ] One excellent example (not multi-language) - [ ] Run scenarios WITH the skill — verify agents comply **REFACTOR — close loopholes:** - [ ] Identify NEW rationalizations from testing - [ ] Add explicit counters (discipline skills) - [ ] Build the rationalization table from all iterations - [ ] Create a red-flags list - [ ] Re-test until bulletproof **Quality:** - [ ] Small flowchart only if the decision is non-obvious - [ ] Quick reference table - [ ] Common mistakes section - [ ] No narrative storytelling - [ ] Supporting files only for tools or heavy reference **Deployment:** - [ ] Commit to git and push (if configured) - [ ] Consider a PR if broadly useful --- ## Package and Present (if `present_files` available) Check whether `present_files` is available. Not available → skip. Available: ```bash python -m scripts.package_skill ``` Point the user to the resulting `.skill` file path for install. ## Principle of Lack of Surprise Skills MUST NOT contain malware, exploit code, or content compromising system security. Skill contents MUST NOT surprise the user in intent if described. Do not create misleading skills, or skills facilitating unauthorized access, data exfiltration, or malicious activity. Roleplay skills are fine. ## Platform Notes **Claude.ai** — no subagents: run test cases yourself, one at a time, and skip baseline runs. No browser → present results directly. Skip quantitative benchmarking, description optimization (needs the `claude` CLI), and blind comparison. Same iteration loop. Packaging works. **Cowork** — subagents available (main workflow works). No browser → `--static `; "Submit All Reviews" downloads `feedback.json`. Packaging works. Description optimization works (`run_loop.py` uses `claude -p`). **Updating an existing skill on either** — preserve the original name. Copy to `/tmp/skill-name/` before editing (the installed path may be read-only). Package from the copy. ## Reference Files Load as needed — do NOT `@`-include (force-loads, burns context). Present in `references/`: - `anthropic-best-practices.md` — Anthropic's official skill authoring guidance; read when writing description fields or structuring a new skill - `testing-skills-with-subagents.md` — complete testing methodology; load when setting up pressure scenarios or baseline runs - `persuasion-principles.md` — behavioral levers for bulletproofing discipline skills; load when writing rationalization tables or red flags - `graphviz-conventions.dot` — dotgraph style conventions; load when adding a flowchart - `render-graphs.js` — render dotgraphs to PNG: `node references/render-graphs.js ` - `examples/CLAUDE_MD_TESTING.md` — worked example Required by the eval harness, NOT present locally — fetch the upstream skill-creator bundle before running it: - `references/schemas.md` — JSON structures (evals.json, grading.json) - `agents/grader.md` · `agents/comparator.md` · `agents/analyzer.md` - `scripts/` · `assets/eval_review.html` · `eval-viewer/generate_review.py`