Skip to content

What is Promptfoo?

Open-source CLI for evaluating and red-teaming LLM apps: compare models side by side, assert on outputs in CI, and scan agents and RAG pipelines for jailbreaks and data leaks.

Pricing
Freemium
Free plan
Yes
Made by
Promptfoo, Inc.
Released
Updated

Compare Promptfoo alternatives

Promptfoo screenshots

Promptfoo features & details

Promptfoo is an open-source tool, MIT-licensed, for testing applications built on large language models. You describe prompts, providers and test cases in a declarative YAML config, run promptfoo eval, and get a matrix of outputs with pass/fail assertions that you can inspect in a local web viewer or in the terminal. It compares models side by side — OpenAI, Anthropic, Gemini, Azure, Bedrock, Ollama and many others, or any HTTP endpoint or Python script — so prompt and model changes are decided on measured results rather than spot checks. Evals run locally, with caching and live reload, and plug into CI/CD pipelines.

The other half is security. promptfoo redteam setup configures an automated red team against your own agent, chatbot or RAG system: it generates application-specific attacks such as direct and indirect prompt injection, jailbreaks, PII and data leaks, insecure tool use and business-rule violations, then produces a vulnerability report grouped by risk category with severity counts. Plugin presets map to frameworks including NIST, OWASP (LLM and API) and MITRE. Code scanning reviews pull requests for LLM-related security issues, and the company also offers guardrails, model security scanning and an MCP proxy.

The command-line tool is free to use with community support and red teaming up to 10,000 probes a month; the Enterprise and on-premise tiers add team sharing, continuous monitoring, compliance dashboards, SSO and managed deployment at custom prices. Install with npm, Homebrew or pip, or run npx promptfoo@latest. In March 2026 Promptfoo announced it had agreed to be acquired by OpenAI, and the project says it remains open source and MIT-licensed.

No reviews yet — sign in to write the first one

Apps similar to Promptfoo

The closest matches are MCPJam and Oqoqo. See every Promptfoo alternative compared by price and free plan.

Featured

Open-source testing and evals platform for MCP servers — trace tool calls, debug OAuth, and catch cross-client regressions in CI.

Coding & DevelopmentFreemium
Featured

Run agent evals and private benchmarks in sandboxed cloud environments — compare Claude Code, Codex, Cursor and Copilot on the same real tasks.

Coding & DevelopmentFreemium

App: Cekura

Tatva Labs Inc.

Featured

Continuous automated QA for voice and chat AI agents, from pre-launch simulation to live call monitoring.

Coding & DevelopmentFreemium

Microsoft's open-source, policy-driven sandbox SDK for running agents, model-generated code, tools and plugins with only the files and network access you grant.

Coding & DevelopmentFree

App: Kane CLI

TestMu AI

Describe a browser or mobile test in plain English and run it from your terminal — Kane CLI drives real Chrome, no selectors or test scripts required.

Coding & DevelopmentFreemium