Evaluation & security
promptfoo/promptfoo
Promptfoo is a testing and red-teaming framework for prompts, models, RAG systems and agents. It lets teams define cases and assertions, compare providers and run regression or security checks in development and CI.
THE PRACTICAL EXPLANATION
Promptfoo is a testing and red-teaming framework for prompts, models, RAG systems and agents. It lets teams define cases and assertions, compare providers and run regression or security checks in development and CI.
WHERE TO USE IT
Use it before releases, during model changes and when testing prompt injection, unsafe behaviour or task-specific quality across a repeatable set of cases.
WHO MAY USE IT
AI engineers, security teams and quality owners who want evaluation expressed as version-controlled test configuration.
HOW TO USE IT
Start with real examples of success and failure, define assertions that reflect the business requirement and compare the current system with the proposed change. Add red-team cases relevant to the connected tools and data, then gate releases on reviewed thresholds.
GETTING THE BEST RESULTS
WHY IT MAY BE USEFUL
Test prompts, agents and RAG systems, including red-team and vulnerability checks.
Best considered for: Regression testing, model comparison and AI security testing in CI.
READ BEFORE YOU ADOPT IT
Test suites only protect scenarios you actually define and maintain.
Confirm the current licence, maintenance status, dependency risk, data path, model access, tool permissions and human approval points at the source. A public repository is inspectable raw material—not proof that a system is secure, supported or suitable for your production environment.