--- name: skill-creator description: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. --- # Skill Creator Audience: AI coding agents first. Create and iteratively improve skills via draft → test → review → repeat. **Process:** 1. Decide skill scope + approach 2. Write draft 3. Create test prompts, run claude-with-skill 4. Evaluate: qualitative + quantitative (use `eval-viewer/generate_review.py`) 5. Rewrite based on feedback + benchmark gaps 6. Repeat until satisfied 7. Expand test set, try again at scale 8. Run description improver to optimize triggering Read user position in this process → jump in, help progress. "I want skill for X" → narrow intent, draft, test, evaluate, repeat. Draft exists → go straight to eval/iterate. ## User Communication Wide range of users. Read context cues. Default: "evaluation"/"benchmark" borderline OK; "JSON"/"assertion" → explain unless user clearly knows them. --- ## Creating a Skill ### Capture Intent Understand user intent first. If conversation contains workflow to capture → extract from history: tools used, step sequence, corrections, input/output formats. Key questions: 1. What should skill enable Claude to do? 2. When should it trigger? (phrases/contexts) 3. Expected output format? 4. Set up test cases? (objective outputs → yes; subjective → decide together) ### Interview and Research Ask about edge cases, input/output formats, example files, success criteria, dependencies. Check available MCPs — research in parallel via subagents if available. Come prepared to reduce burden on user. ### Write the SKILL.md Fill in: - **name**: skill identifier - **description**: when to trigger. Primary trigger mechanism. Include what skill does AND specific contexts. All "when to use" goes here, not in body. Make descriptions "pushy" — include adjacent terms user might say without naming the skill explicitly. - **the rest of the skill** ### Skill Anatomy ``` skill-name/ ├── SKILL.md (required) │ ├── YAML frontmatter (name, description required) │ └── Markdown instructions └── Bundled Resources (optional) ├── scripts/ - Executable code for deterministic/repetitive tasks ├── references/ - Docs loaded into context as needed └── assets/ - Files used in output (templates, icons, fonts) ``` **Progressive Disclosure — 3-level loading:** 1. Metadata (name + description) — always in context (~100 words) 2. SKILL.md body — in context when triggered (<500 lines ideal) 3. Bundled resources — as needed (unlimited; scripts execute without loading) Keep SKILL.md under 500 lines. If approaching limit, add hierarchy with clear pointers. Large reference files (>300 lines) → include table of contents. **Domain organization** for multiple frameworks: ``` cloud-deploy/ ├── SKILL.md (workflow + selection) └── references/ ├── aws.md ├── gcp.md └── azure.md ``` Claude reads only relevant reference file. #### Writing Patterns Use imperative form in instructions. **Output formats:** ```markdown ## Report structure ALWAYS use this exact template: # [Title] ## Executive summary ## Key findings ## Recommendations ``` **Examples pattern:** ```markdown ## Commit message format **Example 1:** Input: Added user authentication with JWT tokens Output: feat(auth): implement JWT-based authentication ``` ### Writing Style Explain WHY behind each instruction — LLMs understand reasoning and go beyond rote instructions. Yellow flag: ALWAYS/NEVER in all caps, rigid structures → reframe with reasoning. Write draft, look fresh, improve. ### Test Cases After draft, write 2-3 realistic test prompts. Share with user: "Here are test cases I'd like to try. Look right?" Then run. Save to `evals/evals.json`: ```json { "skill_name": "example-skill", "evals": [ { "id": 1, "prompt": "User's task prompt", "expected_output": "Description of expected result", "files": [] } ] } ``` See `references/schemas.md` for full schema (including `assertions` field). --- ## Running and Evaluating Test Cases One continuous sequence — don't stop partway. Do NOT use `/skill-test` or any other testing skill. Results go in `-workspace/` as sibling to skill directory. Organize by iteration (`iteration-1/`, etc.), each test case gets directory (`eval-0/`, etc.). Create as you go, not upfront. ### Step 1: Spawn All Runs (with-skill AND baseline) in Same Turn For each test case, spawn two subagents same turn — one with skill, one without. NEVER spawn with-skill first then return for baselines. **With-skill run:** ``` Execute this task: - Skill path: - Task: - Input files: - Save outputs to: /iteration-/eval-/with_skill/outputs/ - Outputs to save: ``` **Baseline run** (same prompt): - New skill → no skill at all, save to `without_skill/outputs/` - Improving existing → old version (snapshot first: `cp -r /skill-snapshot/`), save to `old_skill/outputs/` Write `eval_metadata.json` per test case: ```json { "eval_id": 0, "eval_name": "descriptive-name-here", "prompt": "The user's task prompt", "assertions": [] } ``` ### Step 2: While Runs In Progress, Draft Assertions Draft quantitative assertions per test case. Good assertions: objectively verifiable, descriptive names. Subjective skills → skip assertions, use human judgment. Update `eval_metadata.json` and `evals/evals.json`. Explain to user what they'll see. ### Step 3: As Runs Complete, Capture Timing Save immediately to `timing.json` in run directory: ```json { "total_tokens": 84852, "duration_ms": 23332, "total_duration_seconds": 23.3 } ``` Only opportunity to capture — comes through task notification, not persisted elsewhere. ### Step 4: Grade, Aggregate, Launch Viewer 1. **Grade** — spawn grader subagent (reads `agents/grader.md`). Save to `grading.json` per run. Fields MUST be: `text`, `passed`, `evidence` (NOT `name`/`met`/`details` — viewer depends on exact field names). 2. **Aggregate:** ```bash python -m scripts.aggregate_benchmark /iteration-N --skill-name ``` Produces `benchmark.json` + `benchmark.md`. Put with_skill before baseline counterpart. 3. **Analyst pass** — read `agents/analyzer.md` ("Analyzing Benchmark Results"). 4. **Launch viewer:** ```bash nohup python /eval-viewer/generate_review.py \ /iteration-N \ --skill-name "my-skill" \ --benchmark /iteration-N/benchmark.json \ > /dev/null 2>&1 & VIEWER_PID=$! ``` Iteration 2+: add `--previous-workspace /iteration-`. **No display/headless:** use `--static ` for standalone HTML. Feedback downloads as `feedback.json` on "Submit All Reviews". 5. **Tell user:** "Opened results in browser. 'Outputs' tab: click test cases, leave feedback. 'Benchmark' tab: quantitative comparison. Come back when done." ### Step 5: Read the Feedback ```json { "reviews": [ {"run_id": "eval-0-with_skill", "feedback": "chart missing axis labels", "timestamp": "..."}, {"run_id": "eval-2-with_skill", "feedback": "perfect", "timestamp": "..."} ], "status": "complete" } ``` Empty feedback = user thought it was fine. Focus improvements on specific complaints. Kill viewer when done: `kill $VIEWER_PID 2>/dev/null` --- ## Improving the Skill ### How to Think About Improvements 1. **Generalize** — skill runs millions of times across many prompts. Avoid overfitting to test cases. If stubborn issue, try different metaphors or working patterns. 2. **Keep lean** — remove things not pulling weight. Read transcripts, not just final outputs. 3. **Explain why** — understand what user actually needs and transmit that understanding. ALWAYS/NEVER in all caps → yellow flag → reframe with reasoning. 4. **Bundle repeated work** — all 3 test cases wrote `create_docx.py`? Bundle it as `scripts/create_docx.py`. ### Iteration Loop 1. Apply improvements 2. Rerun all test cases into `iteration-/`, including baselines 3. Launch reviewer with `--previous-workspace` pointing at previous iteration 4. Wait for user to review, say done 5. Read new feedback, improve, repeat Stop when: user says happy, all feedback empty, no meaningful progress. --- ## Advanced: Blind Comparison Rigorous comparison between two skill versions. Read `agents/comparator.md` + `agents/analyzer.md`. Give two outputs to independent agent without labels → judge quality → analyze why winner won. Optional. Human review loop usually sufficient. --- ## Description Optimization After creating or improving skill, offer to optimize description for better triggering accuracy. ### Step 1: Generate Trigger Eval Queries Create 20 eval queries — mix of should-trigger and should-not-trigger: ```json [ {"query": "the user prompt", "should_trigger": true}, {"query": "another prompt", "should_trigger": false} ] ``` Queries MUST be realistic — concrete, specific, with good detail. File paths, personal context, column names, company names, URLs. Mix lowercase/abbreviations/typos/casual. Mix of lengths. Focus on edge cases. - **Should-trigger (8-10):** different phrasings of same intent, cases where user doesn't name skill but clearly needs it - **Should-not-trigger (8-10):** most valuable = near-misses — share keywords but need different skill. Avoid obviously irrelevant negatives. User reviews and signs off. ### Step 2: Review with User 1. Read template from `assets/eval_review.html` 2. Replace `__EVAL_DATA_PLACEHOLDER__` → JSON array, `__SKILL_NAME_PLACEHOLDER__`, `__SKILL_DESCRIPTION_PLACEHOLDER__` 3. Write to `/tmp/eval_review_.html`, open it 4. User edits queries, exports → check `~/Downloads/eval_set.json` ### Step 3: Run Optimization Loop ```bash python -m scripts.run_loop \ --eval-set \ --skill-path \ --model \ --max-iterations 5 \ --verbose ``` Use model ID from system prompt — triggering test must match what user actually experiences. Periodically tail output for user updates. Loop: 60% train / 40% held-out, evaluates each query 3× for reliable trigger rate, proposes improvements, re-evaluates, returns JSON with `best_description`. **How triggering works:** Claude sees `available_skills` list with name + description, decides whether to consult based on description. Complex, multi-step, specialized queries reliably trigger when description matches. Simple one-step queries may not trigger even with matching description — eval queries must be substantive. ### Step 4: Apply Result Take `best_description` from JSON output, update SKILL.md frontmatter. Show before/after, report scores. --- ## Package and Present (if `present_files` available) Check whether `present_files` tool available. If not, skip. If available: ```bash python -m scripts.package_skill ``` Point user to resulting `.skill` file path for install. --- ## Principle of Lack of Surprise Skills must not contain malware, exploit code, or content compromising system security. Skill contents must not surprise user in intent if described. Don't create misleading skills or skills facilitating unauthorized access, data exfiltration, or malicious activity. Roleplay skills are fine. --- ## Claude.ai-Specific No subagents: run test cases yourself (one at a time), skip baseline runs. No browser → present results directly. Skip quantitative benchmarking. Same iteration loop. Skip description optimization (needs `claude` CLI). Skip blind comparison. Packaging works. **Updating existing skill:** preserve original name. Copy to `/tmp/skill-name/` before editing (installed path may be read-only). Package from copy. --- ## Cowork-Specific Subagents available (main workflow works). No browser → `--static `. "Submit All Reviews" downloads `feedback.json` as file. Packaging works. Description optimization works (`run_loop.py` uses `claude -p`). **Updating existing skill:** same as claude.ai section above. --- ## Reference Files `agents/` directory: - `agents/grader.md` — evaluate assertions against outputs - `agents/comparator.md` — blind A/B comparison between two outputs - `agents/analyzer.md` — analyze why one version won `references/`: - `references/schemas.md` — JSON structures (evals.json, grading.json, etc.)