--- name: skill-creator description: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. --- # Skill Creator Skill for creating and iteratively improving skills. High-level process: - Decide what skill does + roughly how - Write draft - Create test prompts, run claude-with-skill on them - Help user evaluate: qualitative + quantitative - While runs happen, draft quantitative evals if none exist. Explain them. - Use `eval-viewer/generate_review.py` to show results. Let user review. - Rewrite skill based on feedback + any glaring benchmark flaws - Repeat until satisfied - Expand test set, try again at scale Figure out where user is in this process, jump in, help progress. If they say "I want skill for X" — narrow down intent, write draft, write test cases, figure out evaluation, run prompts, repeat. If they already have draft — go straight to eval/iterate. Be flexible. If user says "just vibe with me" — do that. After skill done, run description improver (separate script) to optimize triggering. ## Communicating with the user Skill creator used by wide range of users — plumbers, grandparents, and coders alike. Read context cues to calibrate communication. Default: - "evaluation" and "benchmark" — borderline OK - "JSON" and "assertion" — explain unless user clearly knows them Brief definitions OK when in doubt. --- ## Creating a skill ### Capture Intent Understand user intent first. If conversation already contains workflow to capture (e.g., "turn this into a skill"), extract from history: tools used, step sequence, corrections made, input/output formats. User fills gaps, confirms before next step. 1. What should skill enable Claude to do? 2. When should it trigger? (phrases/contexts) 3. Expected output format? 4. Set up test cases? Skills with objectively verifiable outputs (file transforms, data extraction, code gen, fixed workflow steps) benefit from test cases. Subjective outputs (writing style, art) often don't. Suggest default based on skill type, let user decide. ### Interview and Research Ask about edge cases, input/output formats, example files, success criteria, dependencies. Wait to write test prompts until this is ironed out. Check available MCPs — if useful for research, research in parallel via subagents if available, else inline. Come prepared to reduce burden on user. ### Write the SKILL.md Fill in: - **name**: Skill identifier - **description**: When to trigger, what it does. Primary triggering mechanism — include both what skill does AND specific contexts. All "when to use" info goes here, not in body. Make descriptions a bit "pushy" — instead of "How to build dashboard", write "How to build dashboard. Use whenever user mentions dashboards, data visualization, internal metrics, or wants to display company data, even without explicit 'dashboard' request." - **compatibility**: Required tools, dependencies (optional, rarely needed) - **the rest of the skill :)** ### Skill Writing Guide #### Anatomy of a Skill ``` skill-name/ ├── SKILL.md (required) │ ├── YAML frontmatter (name, description required) │ └── Markdown instructions └── Bundled Resources (optional) ├── scripts/ - Executable code for deterministic/repetitive tasks ├── references/ - Docs loaded into context as needed └── assets/ - Files used in output (templates, icons, fonts) ``` #### Progressive Disclosure Three-level loading: 1. **Metadata** (name + description) — Always in context (~100 words) 2. **SKILL.md body** — In context when skill triggers (<500 lines ideal) 3. **Bundled resources** — As needed (unlimited, scripts execute without loading) Word counts approximate — go longer if needed. **Key patterns:** - Keep SKILL.md under 500 lines; if approaching limit, add hierarchy layer with clear pointers - Reference files clearly from SKILL.md with guidance on when to read - For large reference files (>300 lines), include table of contents **Domain organization**: Multiple domains/frameworks — organize by variant: ``` cloud-deploy/ ├── SKILL.md (workflow + selection) └── references/ ├── aws.md ├── gcp.md └── azure.md ``` Claude reads only relevant reference file. #### Principle of Lack of Surprise Skills must not contain malware, exploit code, or content compromising system security. Skill contents must not surprise user in intent if described. Don't create misleading skills or skills facilitating unauthorized access, data exfiltration, or malicious activity. Roleplay skills are fine. #### Writing Patterns Use imperative form in instructions. **Output formats:** ```markdown ## Report structure ALWAYS use this exact template: # [Title] ## Executive summary ## Key findings ## Recommendations ``` **Examples pattern:** ```markdown ## Commit message format **Example 1:** Input: Added user authentication with JWT tokens Output: feat(auth): implement JWT-based authentication ``` ### Writing Style Explain WHY things matter — LLMs are smart, understand reasoning, go beyond rote instructions. If feedback is terse or frustrated, understand the actual task and transmit that understanding. Yellow flag: ALWAYS or NEVER in all caps, super rigid structures — reframe with reasoning instead. Write draft, look with fresh eyes, improve. ### Test Cases After draft, write 2-3 realistic test prompts real users would say. Share with user: "Here are test cases I'd like to try. Look right, or want to add more?" Then run. Save to `evals/evals.json`. No assertions yet — just prompts. Draft assertions next step while runs are in progress. ```json { "skill_name": "example-skill", "evals": [ { "id": 1, "prompt": "User's task prompt", "expected_output": "Description of expected result", "files": [] } ] } ``` See `references/schemas.md` for full schema (including `assertions` field). ## Running and evaluating test cases One continuous sequence — don't stop partway. Do NOT use `/skill-test` or any other testing skill. Results go in `-workspace/` as sibling to skill directory. Organize by iteration (`iteration-1/`, `iteration-2/`, etc.), each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Create directories as you go, not upfront. ### Step 1: Spawn all runs (with-skill AND baseline) in the same turn For each test case, spawn two subagents in same turn — one with skill, one without. Don't spawn with-skill first then come back for baselines. Launch everything at once. **With-skill run:** ``` Execute this task: - Skill path: - Task: - Input files: - Save outputs to: /iteration-/eval-/with_skill/outputs/ - Outputs to save: ``` **Baseline run** (same prompt, baseline depends on context): - **Creating new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`. - **Improving existing skill**: old version. Before editing, snapshot skill (`cp -r /skill-snapshot/`), point baseline subagent at snapshot. Save to `old_skill/outputs/`. Write `eval_metadata.json` for each test case (assertions empty for now). Give descriptive name based on what it tests — not just "eval-0". Use name for directory too. If iteration uses new/modified eval prompts, create these files for each new eval directory. ```json { "eval_id": 0, "eval_name": "descriptive-name-here", "prompt": "The user's task prompt", "assertions": [] } ``` ### Step 2: While runs are in progress, draft assertions Use this time productively. Draft quantitative assertions for each test case, explain to user. If assertions exist in `evals/evals.json`, review and explain them. Good assertions: objectively verifiable, descriptive names — should read clearly in benchmark viewer. Subjective skills (writing style, design) need human judgment — don't force assertions. Update `eval_metadata.json` files and `evals/evals.json` with assertions once drafted. Explain to user what they'll see: qualitative outputs and quantitative benchmark. ### Step 3: As runs complete, capture timing data When each subagent task completes, receive notification with `total_tokens` and `duration_ms`. Save immediately to `timing.json` in run directory: ```json { "total_tokens": 84852, "duration_ms": 23332, "total_duration_seconds": 23.3 } ``` Only opportunity to capture this — comes through task notification, not persisted elsewhere. Process each notification as it arrives. ### Step 4: Grade, aggregate, and launch the viewer Once all runs done: 1. **Grade each run** — spawn grader subagent (or grade inline) that reads `agents/grader.md`, evaluates each assertion against outputs. Save to `grading.json` in each run directory. grading.json expectations array must use fields `text`, `passed`, `evidence` (not `name`/`met`/`details`) — viewer depends on exact field names. For programmatically checkable assertions, write and run script — faster, more reliable, reusable. 2. **Aggregate into benchmark** — run from skill-creator directory: ```bash python -m scripts.aggregate_benchmark /iteration-N --skill-name ``` Produces `benchmark.json` and `benchmark.md` with pass_rate, time, tokens per configuration, mean ± stddev, delta. Put each with_skill version before its baseline counterpart. 3. **Analyst pass** — read benchmark data, surface patterns aggregate stats might hide. See `agents/analyzer.md` ("Analyzing Benchmark Results") — non-discriminating assertions, high-variance evals, time/token tradeoffs. 4. **Launch viewer:** ```bash nohup python /eval-viewer/generate_review.py \ /iteration-N \ --skill-name "my-skill" \ --benchmark /iteration-N/benchmark.json \ > /dev/null 2>&1 & VIEWER_PID=$! ``` Iteration 2+: also pass `--previous-workspace /iteration-`. **Cowork / headless environments:** If `webbrowser.open()` unavailable or no display, use `--static ` for standalone HTML. Feedback downloads as `feedback.json` on "Submit All Reviews". Copy to workspace directory for next iteration. Use `generate_review.py` — no custom HTML needed. 5. **Tell user:** "I've opened results in your browser. Two tabs — 'Outputs' lets you click through test cases and leave feedback, 'Benchmark' shows quantitative comparison. When done, come back and let me know." ### What the user sees in the viewer "Outputs" tab — one test case at a time: - **Prompt**: task given - **Output**: files skill produced, rendered inline where possible - **Previous Output** (iteration 2+): collapsed, last iteration's output - **Formal Grades** (if grading run): collapsed, assertion pass/fail - **Feedback**: textbox that auto-saves - **Previous Feedback** (iteration 2+): prior comments below textbox "Benchmark" tab: pass rates, timing, token usage per configuration, per-eval breakdowns, analyst observations. Navigate via prev/next buttons or arrow keys. "Submit All Reviews" saves feedback to `feedback.json`. ### Step 5: Read the feedback When user says done, read `feedback.json`: ```json { "reviews": [ {"run_id": "eval-0-with_skill", "feedback": "the chart is missing axis labels", "timestamp": "..."}, {"run_id": "eval-1-with_skill", "feedback": "", "timestamp": "..."}, {"run_id": "eval-2-with_skill", "feedback": "perfect, love this", "timestamp": "..."} ], "status": "complete" } ``` Empty feedback = user thought it was fine. Focus improvements on test cases with specific complaints. Kill viewer server when done: ```bash kill $VIEWER_PID 2>/dev/null ``` --- ## Improving the skill Core of the loop. Ran test cases, user reviewed results, now make skill better. ### How to think about improvements 1. **Generalize from feedback.** Creating skills used millions of times across many prompts. You and user iterate on few examples because it moves faster. But skill that only works for those examples is useless. Avoid fiddly overfitty changes or oppressively rigid MUSTs. If stubborn issue, try different metaphors or different working patterns. 2. **Keep prompt lean.** Remove things not pulling weight. Read transcripts, not just final outputs — if skill makes model waste time on unproductive things, cut those parts and see what happens. 3. **Explain the why.** Explain the **why** behind every instruction. LLMs have good theory of mind — when given good harness, go beyond rote instructions. Understand what user actually needs and transmit that understanding into instructions. If writing ALWAYS or NEVER in all caps, or using super rigid structures — yellow flag. Reframe and explain reasoning. More humane, powerful, effective. 4. **Look for repeated work across test cases.** Read transcripts. If all 3 test cases resulted in subagent writing `create_docx.py` or `build_chart.py` — strong signal skill should bundle that script. Write once, put in `scripts/`, tell skill to use it. Take time to really mull things over. Write draft revision, look at it anew, improve. Get into the head of the user. ### The iteration loop After improving skill: 1. Apply improvements 2. Rerun all test cases into new `iteration-/` directory, including baselines. New skill baseline = always `without_skill`. Improving existing skill — use judgment on which baseline makes sense. 3. Launch reviewer with `--previous-workspace` pointing at previous iteration 4. Wait for user to review, say done 5. Read new feedback, improve again, repeat Stop when: - User says happy - All feedback empty - No meaningful progress --- ## Advanced: Blind comparison More rigorous comparison between two skill versions (e.g., "is new version actually better?"). Read `agents/comparator.md` and `agents/analyzer.md`. Basic idea: give two outputs to independent agent without saying which is which, let it judge quality. Analyze why winner won. Optional, requires subagents, most users won't need it. Human review loop usually sufficient. --- ## Description Optimization Description field in SKILL.md frontmatter is primary trigger mechanism. After creating or improving skill, offer to optimize description for better triggering accuracy. ### Step 1: Generate trigger eval queries Create 20 eval queries — mix of should-trigger and should-not-trigger. Save as JSON: ```json [ {"query": "the user prompt", "should_trigger": true}, {"query": "another prompt", "should_trigger": false} ] ``` Queries must be realistic — concrete, specific, with good detail. File paths, personal context, column names, company names, URLs. A bit of backstory. Some lowercase, abbreviations, typos, casual speech. Mix of lengths. Focus on edge cases not clear-cut cases. User gets chance to sign off. Bad: `"Format this data"`, `"Extract text from PDF"`, `"Create a chart"` Good: `"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think"` **Should-trigger queries (8-10):** Different phrasings of same intent — some formal, some casual. Cases where user doesn't name skill but clearly needs it. Uncommon use cases. Cases where skill competes with another but should win. **Should-not-trigger queries (8-10):** Most valuable = near-misses — queries sharing keywords or concepts but needing something different. Adjacent domains, ambiguous phrasing where naive keyword match would trigger but shouldn't, cases touching something skill does but in context where other tool is better. Avoid obviously irrelevant negatives — "Write fibonacci function" for PDF skill tests nothing. Negative cases should be genuinely tricky. ### Step 2: Review with user Present eval set using HTML template: 1. Read template from `assets/eval_review.html` 2. Replace placeholders: - `__EVAL_DATA_PLACEHOLDER__` → JSON array of eval items (no quotes — JS variable assignment) - `__SKILL_NAME_PLACEHOLDER__` → skill's name - `__SKILL_DESCRIPTION_PLACEHOLDER__` → skill's current description 3. Write to temp file (e.g., `/tmp/eval_review_.html`), open: `open /tmp/eval_review_.html` 4. User edits queries, toggles should-trigger, adds/removes entries, clicks "Export Eval Set" 5. Downloads to `~/Downloads/eval_set.json` — check for most recent version if multiple exist This step matters — bad eval queries lead to bad descriptions. ### Step 3: Run the optimization loop Tell user: "This will take some time — I'll run optimization loop in background and check periodically." Save eval set to workspace, then run in background: ```bash python -m scripts.run_loop \ --eval-set \ --skill-path \ --model \ --max-iterations 5 \ --verbose ``` Use model ID from system prompt so triggering test matches what user actually experiences. While running, periodically tail output to give user updates on iteration and scores. Handles full optimization loop automatically: splits eval set 60% train / 40% held-out test, evaluates current description (running each query 3 times for reliable trigger rate), calls Claude to propose improvements based on failures, re-evaluates on both train and test, iterates up to 5 times. Returns JSON with `best_description` — selected by test score to avoid overfitting. ### How skill triggering works Skills appear in Claude's `available_skills` list with name + description. Claude decides whether to consult based on description. Claude only consults skills for tasks it can't easily handle alone — simple one-step queries may not trigger even if description matches, because Claude can handle directly. Complex, multi-step, or specialized queries reliably trigger when description matches. Eval queries must be substantive enough that Claude would actually benefit from consulting skill. Simple queries like "read file X" — poor test cases, won't trigger regardless of description quality. ### Step 4: Apply the result Take `best_description` from JSON output, update skill's SKILL.md frontmatter. Show before/after, report scores. --- ### Package and Present (only if `present_files` tool is available) Check whether `present_files` tool available. If not, skip. If available, package and present: ```bash python -m scripts.package_skill ``` Point user to resulting `.skill` file path for install. --- ## Claude.ai-specific instructions Core workflow same (draft → test → review → improve → repeat), but no subagents changes mechanics: **Running test cases**: No parallel execution. For each test case, read skill's SKILL.md, follow instructions to accomplish test prompt yourself. One at a time. Less rigorous (you wrote skill and run it, full context), but useful sanity check — human review compensates. Skip baseline runs. **Reviewing results**: If no browser available, skip browser reviewer. Present results directly in conversation. For each test case, show prompt and output. If output is file user needs (`.docx`, `.xlsx`), save to filesystem, tell where to download. Ask feedback inline: "How does this look? Anything you'd change?" **Benchmarking**: Skip quantitative benchmarking — relies on baseline comparisons not meaningful without subagents. Focus on qualitative feedback. **The iteration loop**: Same — improve skill, rerun test cases, ask for feedback — just without browser reviewer. Can still organize results into iteration directories. **Description optimization**: Requires `claude` CLI (`claude -p`), only available in Claude Code. Skip on Claude.ai. **Blind comparison**: Requires subagents. Skip. **Packaging**: `package_skill.py` works anywhere with Python and filesystem. User downloads resulting `.skill` file. **Updating existing skill**: - **Preserve original name.** Note skill's directory name and `name` frontmatter — use unchanged. E.g., installed as `research-helper` → output `research-helper.skill`, not `research-helper-v2`. - **Copy to writeable location before editing.** Installed skill path may be read-only. Copy to `/tmp/skill-name/`, edit there, package from copy. - **If packaging manually, stage in `/tmp/` first**, then copy to output — direct writes may fail on permissions. --- ## Cowork-Specific Instructions Key differences: - Subagents available — main workflow (parallel spawning, baselines, grading) works. If severe timeout problems, run test prompts in series. - No browser/display — use `--static ` for standalone HTML. Proffer link user can click. - "Submit All Reviews" downloads `feedback.json` as file. Read from there (may need to request access first). - Packaging works — `package_skill.py` just needs Python and filesystem. - Description optimization (`run_loop.py` / `run_eval.py`) works — uses `claude -p` via subprocess, not browser. Save until skill is finished and user agrees it's in good shape. - **Updating existing skill**: Follow update guidance from claude.ai section above. --- ## Reference files `agents/` directory has instructions for specialized subagents. Read when spawning relevant subagent. - `agents/grader.md` — Evaluate assertions against outputs - `agents/comparator.md` — Blind A/B comparison between two outputs - `agents/analyzer.md` — Analyze why one version beat another `references/` directory: - `references/schemas.md` — JSON structures for evals.json, grading.json, etc. --- Core loop: - Figure out what skill is about - Draft or edit skill - Run claude-with-skill on test prompts - With user, evaluate outputs: - Create benchmark.json and run `eval-viewer/generate_review.py` for user review - Run quantitative evals - Repeat until satisfied - Package final skill, return to user. Add steps to TodoList. If in Cowork, specifically add "Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases" to TodoList. Good luck!