Open protocol and SDKs that connect AI apps to tools and data.
promptfoo
Open-source LLM evals and red teaming.
SharkScore
#42 in Frameworks & SDKs
Why it scores 67.7
promptfoo has a SharkScore of 67.7/100, ranked #42 of 665 in Frameworks & SDKs (updated 2026-11-18).
Each part compares promptfoo with other listings in Frameworks & SDKs. Evidence covers 87% of the score's weight.
- Adoption
- 46 · 30%
- User reviews
- no data · 25%
- Maintenance
- 100 · 20%
- Trust & transparency
- 80 · 15%
- Momentum
- no data · 10%
Paying never changes this score. How ranking works · method v2026-11-18.2
RankShark review of promptfoo
Automated evaluation by RankShark · 18 Nov 2026 · method v1.0 · How we review
We were not paid for this review.
As of 2026-11-18, promptfoo is an open-source LLM evaluation and security-testing tool; its free Community plan includes up to 10k red-team probes per month. [E2, E3]
Promptfoo is a tool for testing prompts, models, RAG pipelines, and agents, with red-teaming and vulnerability-scanning features. It can run locally, and the vendor also offers custom-priced Enterprise and On-Premise plans. [E1, E2, E3]
Best for
- Developers comparing prompts or models and reviewing evaluation results. · docs
- Teams testing RAG pipelines or agent quality. · docs
- Teams seeking automated red-team testing in development workflows. · home page, pricing page
Not for
- Teams that need a published fixed price for Enterprise or On-Premise; both plans are listed as custom-priced. · pricing page
- Teams that need more than 10k red-team probes per month on the free Community plan. · pricing page
Key features
- Prompt and model evaluations. The vendor says evaluations test prompts across models and let users compare outputs in a web viewer. · docs
- RAG and agent evaluation. The vendor documents examples for checking RAG factuality and relevance, and testing agent tool use and response quality. · docs
- Red teaming. The vendor says red teaming generates context-aware attacks, including prompt injections, jailbreaks, data leaks, and insecure tool use. · home page
- Workflow integration. The vendor says it supports CI/CD pipelines, GitHub, GitLab, Jenkins, MCP, and agent frameworks. · home page
- Model file and behavior testing. The vendor says its model security product analyzes model files for risks and tests model behavior against attacks. · security page
Pros
- The free Community plan includes evaluation features, vulnerability scanning, and up to 10k red-team probes per month. · pricing page
- The docs cover prompt, model, RAG, and agent evaluation examples. · docs
- The vendor lists local or self-hosted use for Community and an On-Premise option with complete data isolation. · pricing page
Cons
- Enterprise and On-Premise prices are custom rather than published as fixed amounts. · pricing page
- Community red teaming is limited to 10k probes per month. · pricing page
- The privacy notice says CLI telemetry records command or assertion events unless settings are changed. · privacy page
- Whether customer data is used for model training is not stated on the pages we checked.
Pricing
| Plan | Price | Limits |
|---|---|---|
| Community | Free Forever | All LLM evaluation features; all model providers and integrations; red teaming (10k probes/month); custom integration with your own app; run locally or self-host on your own infrastructure; vulnerability scanning; community support. |
| Enterprise | Custom | All Community features; custom red teaming limits; team sharing & collaboration; continuous monitoring; centralized security/compliance dashboard; customizable attack profiles and target settings; SSO and granular permission profiles; Promptfoo API access; managed cloud deployment; professional services support; priority support & SLA guarantees. |
| On-Premise | Custom | All Enterprise features; deployment on your own infrastructure; complete data isolation; dedicated runner; assigned deployment engineer. |
Checked 18 Nov 2026 · source
- Free plan
- Community is free forever and includes up to 10k red-team probes per month, along with the listed evaluation, integration, local/self-hosting, scanning, and support features. · pricing page
- Free trial
- Not stated (checked 18 Nov 2026)
- Trains on customer data
- Not stated (checked 18 Nov 2026)
- Data retention
- The privacy notice says shared inputs and outputs are stored at the shared location for two weeks. Other retention periods are not stated in the provided evidence. · privacy page
- Data regions
- Not stated (checked 18 Nov 2026)
- Data processing agreement
- Not stated (checked 18 Nov 2026)
- AI models used
- The vendor says it supports 60+ providers, including OpenAI, Anthropic, and Google; it also documents local models such as Ollama. · docs
- API
- Promptfoo API access is listed as an Enterprise feature. · pricing page
- SDKs
- Not stated (checked 18 Nov 2026)
- Webhooks
- Webhooks are listed in the pricing-page feature comparison; plan availability is unclear in the supplied page text. · pricing page
- Platforms
- The docs show setup through npx, npm, or brew, and describe a browser-based setup flow. The pricing page says Community can run locally or be self-hosted. · pricing page, docs
- Security claims
- The vendor says its model file scanner detects malicious code or backdoors, identifies risky model configurations, and flags suspicious operations. · security pageThe vendor says behavioral testing can test models against jailbreaks and injections and simulate real-world attacks. · security pageThe vendor says its model-security product supports PyTorch, TensorFlow, Keras, Pickle, and JSON/YAML files. · security page
What we measured
- GitHub stars · now
- 27,395
- Latest release · now
- 0.125.1 on 2026-10-25
- Package downloads · 30 days
- 4,709,923
- Tranco traffic rank · list of the day
- #83,491
How we tested · sources
- Hourly uptime and response-time checks of the product's URL.
- Repository, release and download data from GitHub, npm, PyPI and Hugging Face where they exist.
- We did not sign up, pay, or use the product as a customer.
- home page: https://www.promptfoo.dev/ · fetched 18 Nov 2026
- pricing page: https://www.promptfoo.dev/pricing/ · fetched 18 Nov 2026
- docs: https://www.promptfoo.dev/docs/getting-started/ · fetched 18 Nov 2026
- changelog: https://www.promptfoo.dev/docs/releases/ · fetched 18 Nov 2026
- privacy page: https://www.promptfoo.dev/privacy/ · fetched 18 Nov 2026
- terms: https://www.promptfoo.dev/terms-of-service/ · fetched 18 Nov 2026
- security page: https://www.promptfoo.dev/model-security/ · fetched 18 Nov 2026
Something wrong? Report an error with a public link that shows it; corrections re-run the review.
Top alternatives to promptfoo
See all- 81.3ToolOpen source
- 77.7
TypeScript toolkit for building AI apps and agents.
ToolOpen source - 77.7
Build agents and LLM apps in Python and TypeScript.
ToolOpen source - 76.9
Tracing, evaluation and monitoring for LLM apps.
ToolFreemium · from $39/mo