Skip to content
SkillHone

SkillHoneSkill

by Tencent
Released
22 views
MIT
Repository Docs
Featured markdown Development

Summary

Tencent's skill-evolution harness: it rewrites a whole skill folder — SKILL.md, scripts and references together — and lands every decision as a real Git issue, PR and wiki entry you can review.

Features

  • Optimises the whole skill folder — SKILL.md, scripts/ and references/ in one atomic PR
  • Every decision lands as a real Git issue, branch, commit, PR or wiki entry
  • Runs on a local Forgejo instance, so the audit trail never leaves your machine
  • Hard eval/skill split enforced by code paths and filesystem permissions, not prompt convention
  • Ships as standard agentskills.io skills — no per-runtime adapter to maintain
  • One LiteLLM gateway config spans Anthropic, OpenAI, DeepSeek, Gemini and more
  • Separate credentials and endpoints for the improver, executor and synthesis stages

Install This Skill

Add this skill to your favorite AI agent in a few steps.

Any AI agent

This skill is plain instructions — it works with any assistant that accepts custom instructions or system prompts.

  1. Copy the skill content with the button below.
  2. Paste it into your agent's instruction file or system prompt (for example AGENTS.md, .cursorrules, or a custom instructions field).
  3. Ask the agent to apply the skill whenever the task matches.

Skill Content

Markdown Content

Copy this content and use it with your preferred AI agent

---
name: skillhone
description: >
  SkillHone — toolkit for evaluating, optimizing, and managing agent skills.
  Use when asked to "evaluate a skill", "run probe", "optimize/iterate a skill",
  "create a new skill experiment", "seed a skill repo", or "run skill benchmarks";
  also use when the user mentions a Forgejo-hosted skill repo and wants to
  measure or improve its quality. Wraps standalone scripts: status, eval,
  optim, new, seed, serve, synth.
compatibility: Requires Python 3.10+, git, and access to a Forgejo instance (or local FS).
---

# SkillHone

SkillHone is a measurement harness plus an optimization toolkit for agent
skills. The important abstraction is not a fixed workflow; it is the evidence
trail created when a skill is run against private eval tasks.

The harness has these layers:

- **Skill repo**: public behavior to improve (`SKILL.md`, scripts, references).
- **Eval repo**: private measurement contract (datasets, verifier, synthesis
  contract, task-local validators).
- **Solver workdirs**: isolated per-item sandboxes containing artifacts and
  `trajectory.jsonl`.
- **Observation records**: redacted probe results, trajectory diagnosis,
  compiler/validator diagnosis, issues, PRs, and wiki pages.

When improving a skill, identify which layer explains the failure before
changing code. A score drop may point to skill instructions, but it can also be
a harness, verifier, compiler, artifact-path, or infrastructure problem.

Standalone scripts live under `scripts/`. Pick one based on the task in front of you.

Before starting a diagnosis, development, PR review, merge, or optimization cycle on a Forgejo-backed repo, run `scripts/status.py` so you know the current issue/PR state and do not duplicate work or merge the wrong PR.

## Eval Synthesis Contract

When creating or synthesizing eval data, the task spec / README is the source of
truth for both prompts and verification. Do not generate verifiers that only
check the gold answer if the task spec also requires observable output
properties.

If the README says the answer must have a format, file, syntax, parser/compile
success, render success, required section, count range, banned token, fixed
style token, palette, local-only dependency, or other deterministic acceptance
criterion, include a corresponding `scores` key in the verifier. This applies
whether you use `skillhone-synthesis` or write a small generation script by
hand.

Subjective requirements may be approximated by deterministic proxies. Truly
uncheckable preferences should be noted as unverified, not silently ignored.

## Scripts

| Script | Use when the user asks to... | Details |
|--------|-------------------------------|---------|
| `scripts/status.py` | inspect the current Forgejo repo's issue/PR state before acting | [references/cli.md](references/cli.md) |
| `scripts/eval.py` | run a probe / test evaluation on a skill, get a score | [references/evaluation.md](references/evaluation.md) |
| `scripts/optim.py` | iterate on / optimize / auto-improve a skill | [references/optim.md](references/optim.md) |
| `scripts/new.py` | create a new skill+eval repo pair (a new experiment) | [references/cli.md](references/cli.md) |
| `scripts/seed.py` | scaffold SKILL.md / scripts/ for an empty skill repo | [references/seed.md](references/seed.md) |
| `scripts/synth.py` | synthesise a probe split into the eval repo (calls into `skillhone-synthesis`) | [references/cli.md](references/cli.md) |
| `scripts/serve.py` | listen for Forgejo webhooks (PR/issue/push) | [references/cli.md](references/cli.md) |
| `scripts/quality/static_check.py` | validate skill structure and script discoverability | [references/quality_scoring_rubric.md](references/quality_scoring_rubric.md) |

Load the relevant reference file only when you're about to run that specific script. Do not load them up front.

## Upstream binding & sync

The `skillhone` skill is bound to **<https://github.com/Tencent/SkillHone>**;
the repository is the actual implementation. When the skill is delivered
through a hub (SkillHub, an internal mirror, any `agentskills.io`-protocol
runtime), the SKILL.md alone is not enough — the runtime has to pull the
full upstream repository before the harness scripts can run.

Follow [`references/upstream.md`](references/upstream.md) whenever you need to:

- **Install for the first time** — `~/.skillhone/skills/` is empty.
- **Refresh / update / pull latest** — refresh existing skill folders.

Both run the same idempotent procedure (shallow clone + `cp -R` into
`~/.skillhone/skills/<skill>/`). Settings, run history, and the cache stay
untouched.

## Quick examples

```bash
# Status — read-only dashboard of Issues + PRs for the current Forgejo repo
python3 scripts/status.py

# Evaluate — runs probe split, writes result.json
python3 scripts/eval.py --skill-dir ./my-skill --eval-dir ./my-skill-eval \
                        --split probe --output result.json

# Optimize — agent-driven loop (5 iters, stop after 2 with no gain)
python3 scripts/optim.py --repo http://forgejo/skillhone/my-skill.git \
                         --iters 5 --patience 2

# New experiment — creates skill-repo + eval-repo on Forgejo
python3 scripts/new.py deep-research \
                       --instruction README.md --data-dir ./data --no-run

# Seed an empty skill — generate SKILL.md from a brief
python3 scripts/seed.py --repo http://forgejo/skillhone/my-skill.git

# Webhook listener
python3 scripts/serve.py --port 8790
```

## Gotchas

- **`~/.skillhone/settings.json` is required** before any script runs. It holds the Forgejo URL/token and three model profiles: `improver` (drives `optim.py`), `executor` (runs the skill under eval), and optional `synthesis` (used by `synth.py`). See [references/configuration.md](references/configuration.md).
- **Start with `scripts/status.py` on Forgejo repos.** It is read-only and shows the open/closed Issue and PR state for the current repo; use it before creating issues, developing fixes, reviewing PRs, or merging.
- **`scripts/eval.py` never writes to the skill repo** — it reads SKILL.md and writes a JSON result. Safe to run read-only.
- **`scripts/optim.py` spawns subagents** (issue-reporter → developer → reviewer) via the Agent tool. Do *not* add `Agent` / `Task` to `disallowed_tools` in `settings.json.improver` or the loop will no-op.
- **Eval repo must stay private.** The optimizing agent must never see it. `optim.py` only passes the skill repo path + a redacted probe result into the loop.
- **Probe ≠ test.** A probe improvement does not guarantee a test improvement. See `references/evaluation.md` → "Probe vs Test".
- **All state lives under `~/.skillhone/`** (logs, run artifacts, workspaces). Override with `$SKILLHONE_HOME`.

## How this skill composes

```
skillhone-prd   →  skillhone-synthesis  →  skillhone  →  skillhone-evaluation  →  skillhone-optimization
(spec the PRD)     (generate eval data)    (eval/optim)    (score + diagnose)       (optimize via PR)
```

`skillhone` is the orchestrator entry point; evaluation and optimization skills
are loaded on demand inside `optim.py`'s agent loop. VCS operations are provided
by a separate backend skill such as `forgejo`.

## Orchestration: "synthesise and optimise a skill from <PRD>"

When the user asks to "synthesise and optimise a skill from `<path/to/PRD.md>`"
(typical phrasing for the worked examples under `examples/`), run the four
scripts below in order. The contract is one persistent skill repo + one
private eval repo on Forgejo, with a regression-aware synth step gating the
expensive optim phase.

1. **`scripts/new.py <skill-name> --instruction <PRD.md>`**
   Creates the public `<skill-name>` and private `<skill-name>-eval` repos
   on Forgejo. Auto-redacts the PRD's `## ...Evaluation/Verifier/Scoring/
   Rubric...` section so the public README never exposes the grading rubric
   to the improver. The unredacted PRD lands in the eval repo.

2. **`scripts/seed.py --repo <skill-url>`**
   Reads the redacted public README, generates a real (but unoptimised)
   `SKILL.md` plus minimal scaffolding, and commits as the seed point. This
   is the baseline the synth-stage regression scores against — without a
   real seed, the regression is meaningless. Skip this step ONLY when the
   PRD has no `## 3.5 Synth-stage acceptance gate` and you are intentionally
   running an old-style single-shot synth.

3. **`scripts/synth.py --repo <skill-url> --target 10 --splits probe ...`**
   Synthesises `probe.jsonl` from the eval-side PRD. When the PRD declares
   a synth-stage acceptance gate (§3.5 in the worked examples), pass
   `--target-pass-rate-max <X> --max-resynth <N>` (typical: `0.30` and `3`)
   so synth runs `eval.py --mode seed --split probe` after each draft and
   redrafts when the seed solves more than X of the probes. Each iteration's
   observations are written to the eval repo's `synthesis_observations/`
   directory and pushed alongside the final `probe.jsonl`. Without these
   flags, synth is single-shot (the historical behaviour).

   > **Synth is optional.** If you already have a curated eval set (golden
   > items from a benchmark, hand-written probes, an exported test bank,
   > etc.), skip `synth.py` entirely and push your own `probe.jsonl`
   > (and optionally `test.jsonl`) directly into the eval repo. The
   > format the rest of the harness expects is documented in
   > [`references/evaluation.md`](references/evaluation.md). As long as
   > the verifier contract is satisfied, `optim.py` does not care whether
   > the data came from `synth.py` or `git push`.

4. **`scripts/optim.py --repo <skill-url> --iters 3 --patience 2`**
   The agent-driven PR loop: diagnose probe failures → file Issue → land
   focused PR → re-evaluate → write `Iteration-N-Observation` wiki page.
   Each merged PR is one atomic skill change.

Skip steps 2 + 3's `--target-pass-rate-max` only for prototype runs where you
explicitly want to see what synth produces without a regression gate. For any
example whose PRD includes §3.5, skipping the gate defeats the point.

## References (load on demand)

- [references/evaluation.md](references/evaluation.md) — `eval.py` CLI, output JSON schema, solver architecture. Read before running `eval.py` or interpreting `result.json`.
- [references/optim.md](references/optim.md) — `optim.py` loop, subagent roles, stop conditions. Read before running `optim.py`.
- [references/seed.md](references/seed.md) — original SkillHone seed scaffold and validation rules. Read before running `seed.py`.
- [references/quality_scoring_rubric.md](references/quality_scoring_rubric.md) — rubric used by quality reviewers.
- [references/configuration.md](references/configuration.md) — `~/.skillhone/settings.json` schema, directory layout, env vars. Read on first setup or when the user asks "where do I configure X?".
- [references/cli.md](references/cli.md) — flag-by-flag reference for every script. Read when a user asks about a flag you're not sure about.

Usage Instructions

Learn how to use this skill with different AI agents.

Generic Instructions

Paste this into any skill-capable assistant (Claude Code, Codex, OpenClaw, Hermes):

Please install SkillHone by following the instructions at
https://raw.githubusercontent.com/Tencent/SkillHone/main/docs/install/skillhone.md.
Detect my agent runtime, install the skillhone skill into its skills
directory, and then ask me for the model credentials needed to finish
configuration.

Run it only inside a sandbox, container, VM or disposable clone — validation workflows execute local commands.

Example Usage

/skillhone optimize my travel-qa skill for 5 iterations, then evaluate it against the latest probe split.

Description

Most "self-improving skill" projects hand an LLM a SKILL.md string and ask it to write a better one. SkillHone, from Tencent, changes both the unit of work and the record of it.

The unit of change is the whole skill folder. A single merged pull request can rewrite SKILL.md, add a helper under scripts/, and drop a new reference page under references/ as one atomic change, gated by a regression suite. Prompt-only optimisers cannot express that, which matters once a skill's behaviour depends on the code it shells out to as much as on its instructions.

The record is a real Git server. Every step in the loop lands as an issue, branch, commit, pull request or wiki entry on a Forgejo instance that can run entirely on your own machine. When you want to know why a skill changed, you open a UI reviewers already know how to read and the whole decision path is sitting there — no bespoke log format to learn.

What holds it together
  • A hard eval/skill split enforced by code paths and filesystem permissions rather than prompt convention, so held-out probe tasks are much harder to leak into the skill's own instructions by accident.
  • No runtime adapter. SkillHone ships as a bundle of standard agentskills.io skills, so any runtime that already supports skills supports it — Claude Code, Codex, OpenClaw, Hermes, and whatever comes next.
  • A unified LiteLLM gateway (August 2026) means one provider/model string covers Anthropic, OpenAI, DeepSeek, Gemini and other providers, and the improver, executor and synthesis stages can each hold independent credentials and endpoints.
Installing and running it

Installation is conversational: paste the install prompt into any skill-capable assistant and it fetches the guide, detects your runtime, and places the skillhone skill correctly. Then invoke it like any skill — /skillhone optimize my travel-qa skill for 5 iterations — and the top-level skill dispatches to the right sub-skill. A companion skillhone-prd sub-skill drafts a PRD for a new skill before the first optimisation pass.

One caution worth heeding: some workflows use bypass mode and local exec/subprocess calls during validation. Run it in a sandbox, container, VM or disposable clone — not in a directory holding credentials, production data or unrelated repositories.

Requires Python 3.10+, git, and a Forgejo instance or local filesystem. MIT licensed; the method is written up in arXiv 2606.08671.

Related Skills

Skill: Inngest Agents

by Inngest

New

Build crash-safe AI agents on AgentKit and step.ai — durable tool calls, human-in-the-loop approval, realtime progress, and when not to reach for an agent loop at all.

Development
2 views

Skill: Inngest Durable Functions

by Inngest

New

Inngest's durable-execution playbook as an agent skill — steps and memoisation, event and cron triggers, idempotency, cancellation, retries and non-retriable errors.

Development
2 views

Skill: Temporal Developer

by Temporal

New

Temporal's official skill for building durable workflows — SDK patterns across seven languages, plus the determinism rules that decide whether a workflow survives a replay.

Development
7 views
Browse all skills →