Skip to content
RankShark

promptfoo

Open-source LLM evals and red teaming.

ToolOpen sourceby Promptfoosince 2023
67.7

SharkScore

#42 in Frameworks & SDKs

Why it scores 67.7

promptfoo has a SharkScore of 67.7/100, ranked #42 of 665 in Frameworks & SDKs (updated 2026-11-18).

Each part compares promptfoo with other listings in Frameworks & SDKs. Evidence covers 87% of the score's weight.

Adoption
46 · 30%
User reviews
no data · 25%
Maintenance
100 · 20%
Trust & transparency
80 · 15%
Momentum
no data · 10%

Paying never changes this score. How ranking works · method v2026-11-18.2

RankShark review of promptfoo

Automated evaluation by RankShark · 18 Nov 2026 · method v1.0 · How we review

We were not paid for this review.

As of 2026-11-18, promptfoo is an open-source LLM evaluation and security-testing tool; its free Community plan includes up to 10k red-team probes per month. [E2, E3]

Promptfoo is a tool for testing prompts, models, RAG pipelines, and agents, with red-teaming and vulnerability-scanning features. It can run locally, and the vendor also offers custom-priced Enterprise and On-Premise plans. [E1, E2, E3]

Best for

  • Developers comparing prompts or models and reviewing evaluation results. · docs
  • Teams testing RAG pipelines or agent quality. · docs
  • Teams seeking automated red-team testing in development workflows. · home page, pricing page

Not for

  • Teams that need a published fixed price for Enterprise or On-Premise; both plans are listed as custom-priced. · pricing page
  • Teams that need more than 10k red-team probes per month on the free Community plan. · pricing page

Key features

  • Prompt and model evaluations. The vendor says evaluations test prompts across models and let users compare outputs in a web viewer. · docs
  • RAG and agent evaluation. The vendor documents examples for checking RAG factuality and relevance, and testing agent tool use and response quality. · docs
  • Red teaming. The vendor says red teaming generates context-aware attacks, including prompt injections, jailbreaks, data leaks, and insecure tool use. · home page
  • Workflow integration. The vendor says it supports CI/CD pipelines, GitHub, GitLab, Jenkins, MCP, and agent frameworks. · home page
  • Model file and behavior testing. The vendor says its model security product analyzes model files for risks and tests model behavior against attacks. · security page

Pros

  • The free Community plan includes evaluation features, vulnerability scanning, and up to 10k red-team probes per month. · pricing page
  • The docs cover prompt, model, RAG, and agent evaluation examples. · docs
  • The vendor lists local or self-hosted use for Community and an On-Premise option with complete data isolation. · pricing page

Cons

  • Enterprise and On-Premise prices are custom rather than published as fixed amounts. · pricing page
  • Community red teaming is limited to 10k probes per month. · pricing page
  • The privacy notice says CLI telemetry records command or assertion events unless settings are changed. · privacy page
  • Whether customer data is used for model training is not stated on the pages we checked.

Pricing

PlanPriceLimits
CommunityFree ForeverAll LLM evaluation features; all model providers and integrations; red teaming (10k probes/month); custom integration with your own app; run locally or self-host on your own infrastructure; vulnerability scanning; community support.
EnterpriseCustomAll Community features; custom red teaming limits; team sharing & collaboration; continuous monitoring; centralized security/compliance dashboard; customizable attack profiles and target settings; SSO and granular permission profiles; Promptfoo API access; managed cloud deployment; professional services support; priority support & SLA guarantees.
On-PremiseCustomAll Enterprise features; deployment on your own infrastructure; complete data isolation; dedicated runner; assigned deployment engineer.

Checked 18 Nov 2026 · source

Free plan
Community is free forever and includes up to 10k red-team probes per month, along with the listed evaluation, integration, local/self-hosting, scanning, and support features. · pricing page
Free trial
Not stated (checked 18 Nov 2026)
Trains on customer data
Not stated (checked 18 Nov 2026)
Data retention
The privacy notice says shared inputs and outputs are stored at the shared location for two weeks. Other retention periods are not stated in the provided evidence. · privacy page
Data regions
Not stated (checked 18 Nov 2026)
Data processing agreement
Not stated (checked 18 Nov 2026)
AI models used
The vendor says it supports 60+ providers, including OpenAI, Anthropic, and Google; it also documents local models such as Ollama. · docs
API
Promptfoo API access is listed as an Enterprise feature. · pricing page
SDKs
Not stated (checked 18 Nov 2026)
Webhooks
Webhooks are listed in the pricing-page feature comparison; plan availability is unclear in the supplied page text. · pricing page
Platforms
The docs show setup through npx, npm, or brew, and describe a browser-based setup flow. The pricing page says Community can run locally or be self-hosted. · pricing page, docs
Security claims
The vendor says its model file scanner detects malicious code or backdoors, identifies risky model configurations, and flags suspicious operations. · security pageThe vendor says behavioral testing can test models against jailbreaks and injections and simulate real-world attacks. · security pageThe vendor says its model-security product supports PyTorch, TensorFlow, Keras, Pickle, and JSON/YAML files. · security page

What we measured

GitHub stars · now
27,395
Latest release · now
0.125.1 on 2026-10-25
Package downloads · 30 days
4,709,923
Tranco traffic rank · list of the day
#83,491
How we tested · sources
  • Hourly uptime and response-time checks of the product's URL.
  • Repository, release and download data from GitHub, npm, PyPI and Hugging Face where they exist.
  • We did not sign up, pay, or use the product as a customer.

Something wrong? Report an error with a public link that shows it; corrections re-run the review.

Top alternatives to promptfoo

See all
  • 81.3

    Open protocol and SDKs that connect AI apps to tools and data.

    ToolOpen source
  • 77.7

    TypeScript toolkit for building AI apps and agents.

    ToolOpen source
  • 77.7

    Build agents and LLM apps in Python and TypeScript.

    ToolOpen source
  • 76.9

    Tracing, evaluation and monitoring for LLM apps.

    ToolFreemium · from $39/mo