Skip to content
SerpApi Agent Usability Test

SerpApi Agent Usability TestSkill

by SerpApi
Added to Onei
1 views
v1.0
MIT
Repository
markdown Development

Summary

A protocol for measuring whether agents can discover and use your tool without being told about it — WITH/WITHOUT trials, binary scoring, and lift as the only metric. The subject under test is your docs, not the model.

Features

  • Lift (WITH minus WITHOUT) as the single metric, with the WITHOUT baseline treated as mandatory
  • Five named failure modes, each pointing at a fixable part of the interface
  • Isolation guidance that uses a stripped HOME, plus a check that the isolation actually held
  • Randomised run order to avoid temporal confounds from rate limits and price changes
  • Binary per-fact scoring with pass/fail criteria defined before the run — no rubrics, no borderline judgement
  • Discovery-rate baselines by integration level, so a file-on-disk test of an MCP tool is read as a false negative
  • Pre-flight checklist including an information-asymmetry check that predicts zero lift for easy tasks
  • Adversarial conditions (429, timeout, malformed response, deprecated endpoint) that test error messages

Install This Skill

Add this skill to your favorite AI agent in a few steps.

Any AI agent

This skill is plain instructions — it works with any assistant that accepts custom instructions or system prompts.

  1. Copy the skill content with the button below.
  2. Paste it into your agent's instruction file or system prompt (for example AGENTS.md, .cursorrules, or a custom instructions field).
  3. Ask the agent to apply the skill whenever the task matches.

Skill Content

Markdown Content

Copy this content and use it with your preferred AI agent

---
name: agent-usability-test
description: >-
  Test whether agents can discover and use your tool — not whether agents are
  capable. The subject under test is the interface. A bad score means fix the
  docs/tool, not the agent.
license: MIT
version: "1.0"
---

Give agent a goal. Make tool available. Don't mention the tool. Observe.
Run WITHOUT baseline. Delta = **lift** — the only metric that matters.
Lift is inversely proportional to model capability: docs matter most for weak models; strong models self-correct from API responses.

## Failure modes

| # | Failure | Signal |
|---|---------|--------|
| 1 | Non-discovery | Tool never called despite being available and relevant |
| 2 | Wrong selection | Agent picks suboptimal tool when multiple are available |
| 3 | Parameter cargo-culting | Agent copies doc examples instead of adapting |
| 4 | Response-schema blindness | Correct call, wrong field extracted |
| 5 | Auth/error cliff | 401/429/timeout → agent gives up instead of recovering |

## Discovery by integration level

```
MCP tool registered    ~100%    (in agent's tool list)
System prompt hint     ~50-80%  (estimate)
CLI on $PATH           ~30-50%  (N=4)
File on disk           0%       (N=12, 4 models)
```

Test at your deployment level. File-on-disk test for MCP-deployed tool = false negative.

## Protocol

1. **Hypothesize.** State expected outcome before running. Fisher's exact for N<20.
2. **Design tasks** with verifiable answers (binary: correct/incorrect). Don't encode methodology in the prompt.
3. **Verify ground truths same-day.** Query the source yourself before running trials. Stale GTs produce false negatives.
4. **Run matrix.** ≥2 models × 2 conditions (WITH/WITHOUT). Uncoached prompt: *"Answer this: [GOAL]. Cite your source."*
5. **Randomize run order.** Shuffle all (model, task, condition, trial) tuples. Sequential runs introduce temporal confounds (API rate limits, model load, price changes).
6. **Isolate WITHOUT completely.** HOME controls skill/extension loading in most agent CLIs — don't just set CWD=/tmp. Create a stripped HOME with agent auth config but no skills directory, no extensions. Tool off PATH. No env vars. Verify: ask the WITHOUT agent "what skills do you have?" — if it names your tool, isolation failed. If it names your tool despite correct isolation, that's training-data contamination — note it, don't fix it.
7. **Competition variant (FM#2):** Give ALL competing tools simultaneously. Score which gets picked. Three conditions: YOURS-ONLY, ALL-TOOLS, NONE.
8. **Observe via trace** — not self-report. Metrics: discovery rate, selection rate, efficiency (calls to correct answer), recovery rate, lift.
9. **Score binary per fact.** Automated substring or exact match. No 0-100 rubrics. No human judgment on borderline cases (define pass/fail criteria before running).
10. **Fix → Retest with control.** Fix docs, not agent. Run old-docs AND new-docs agents in same session — without a control, improvement could be model variance.

## Adversarial conditions (tests your error messages, not agent intelligence)

- **429 rate limit:** retry with backoff or give up?
- **Network timeout:** fall back or fail silently?
- **Malformed response:** handle unexpected JSON shape?
- **Deprecated endpoint:** find current one from error message?

Score: binary (recovered / didn't).

## Don't

- Coach the agent ("use this tool") — tests reading, not behavior
- Ask agents to self-report friction
- Test one model only
- Skip the WITHOUT baseline
- Use 0-100 rubric scores
- Claim significance at N<10
- Score with stale ground truths (verify same-day)
- Run WITHOUT from the repo directory (AGENTS.md leaks tool names)
- Run all WITH then all WITHOUT sequentially (randomize)
- Report behavioral observation ("used the tool") as score data ("correct answer")

## Sample size

N=1-3/cell → directional only (never publish) · N=10/cell → Fisher's exact, large effects · N=12/cell → 80% power, moderate effects · Always report N per cell, not total runs

## Pre-flight checklist (run before committing to full matrix)

1. [ ] GTs verified same-day via the tool itself (not from memory/training data)
2. [ ] Pilot: 1 run WITH — agent discovers and uses the tool? If not, fix infra.
3. [ ] Pilot: 1 run WITHOUT — agent has zero awareness of tool? If not, fix isolation.
4. [ ] Information asymmetry check: can web search answer this task? If yes, expect 0pp lift regardless of skill quality. Test with 1 WITHOUT run — if correct, the task is too easy.
5. [ ] Scoring function matches GT format (comma-separated numbers, decimal points, currency symbols)
6. [ ] Error rate <10% in pilot (rate limits, auth failures, timeouts)
7. [ ] Questions span ≥3 engines/capabilities (not all the same difficulty)

## Not this

| Approach | Tests | Subject |
|----------|-------|---------|
| WebBench/WebArena | Can agent complete web tasks? | Agent capability |
| API-Bank/ToolBench | Can agent follow API specs? | Agent tool-use skill |
| UXAgent/UXCascade | Is web UI usable for humans? | Human-facing interface |
| Search API benchmarks | Which API gives better results? | API output quality |
| **AUT** | **Can agents discover and use it?** | **Agent-facing interface** |

Usage Instructions

Learn how to use this skill with different AI agents.

Generic Instructions
npx skills add https://github.com/serpapi/skills --skill agent-usability-test

Or copy skills/agent-usability-test/SKILL.md from the repository into your agent's skills directory. No API key required — the protocol is tool-agnostic.

Example Usage

Run an agent usability test on our MCP server: design five tasks with verifiable answers, run WITH and WITHOUT across two models, and report the lift with N per cell.

Description

Everyone shipping an MCP server or a CLI for agents eventually asks the same question and answers it badly: does this work for agents? The usual test is to tell an agent about the tool and watch it succeed, which measures nothing. Agent Usability Test (AUT) is SerpApi's published protocol for measuring the thing that actually matters — whether an agent finds and uses your interface when nobody mentions it — and it is written as a runnable experimental design, not a blog post.

The premise

Give the agent a goal. Make the tool available. Say nothing about it. Observe. Run the same tasks without the tool present, and the difference — lift — is the only metric. The skill's sharpest observation is that lift is inversely proportional to model capability: strong models self-correct from API responses, so your documentation matters most for the weak ones.

Five failure modes, named

Non-discovery (never called despite being relevant), wrong selection (a worse tool wins when several are available), parameter cargo-culting (the agent copies your doc examples instead of adapting them), response-schema blindness (right call, wrong field extracted), and the auth/error cliff (a 401, 429 or timeout ends the attempt instead of triggering recovery). Each maps to something you can fix in the interface.

Methodology that survives scrutiny

The ten-step protocol is unusually strict for a vendor-published document: hypothesise before running; design tasks with binary verifiable answers; verify ground truths the same day, because stale ones produce false negatives; run a matrix of at least two models × two conditions; randomise run order, since sequential WITH-then-WITHOUT runs pick up rate limits and price changes as confounds; isolate the WITHOUT condition properly — a stripped HOME, not just a different working directory, because HOME is what loads skills in most agent CLIs — and verify the isolation by asking the agent what skills it has; observe via trace rather than self-report; score binary per fact with no 0–100 rubrics; and retest with a control, because an improvement without one could be model variance.

It also fixes discovery expectations to integration level — MCP registration ≈100%, a system-prompt hint 50–80%, a CLI on $PATH 30–50%, a file on disk 0% — so a file-on-disk test of an MCP-deployed tool is correctly called a false negative.

The parts most teams skip

A pre-flight checklist with an information-asymmetry check: if plain web search can answer your task, expect zero lift no matter how good the skill is. Sample-size guidance (N=1–3 is directional and should never be published; N=12/cell for 80% power on moderate effects). Adversarial conditions that test your error messages — 429, timeout, malformed response, deprecated endpoint — scored simply as recovered or not. And an explicit "Don't" list: no coaching the agent, no self-reported friction, no single-model tests, no significance claims below N=10.

Who it is for

Anyone publishing an agent-facing interface — MCP server, CLI, SDK or skill — who wants evidence rather than a demo. MIT licensed, version 1.0, maintained by SerpApi.

Related Skills

Skill: Trail of Bits Static Analysis

by Trail of Bits

New

CodeQL, Semgrep and SARIF static-analysis toolkit from Trail of Bits: taint tracking, fast pattern scans and merged, deduplicated security findings for coding agents.

Development
3 views

Skill: Playwright CLI

by Microsoft

New

Microsoft's official Playwright skill — drives a real browser from the command line using accessibility snapshots and element refs, and plans, generates and heals Playwright tests.

Development
7 views 1 copies

Skill: Google Maps Platform

by Google Maps Platform

New

Google's official agent skill for writing production Maps Platform code — grounded in freshly fetched docs, with a demo key path that needs no billing account.

Development
8 views

Skill: Use Railway

by Railway

New

Railway's official agent skill: create projects, provision databases and buckets, deploy, manage variables and domains, and read build failures back — from the CLI, API or MCP server.

Development
8 views
Browse all skills →