Promptfoo — Test, Evaluate & Red-Team Your AI Apps
Test prompts, evaluate AI responses, and identify security risks before they affect real users. Promptfoo is an open-source testing framework that helps developers build more reliable AI-powered applications through structured evaluations, automated checks, and red teaming.
Whether you're testing a chatbot, refining an AI agent, or checking the quality of generated responses, Promptfoo provides a practical way to measure performance, compare results, and uncover weaknesses.
Free Open-Source Tool | Windows, macOS & Linux | Command-Line Installation
What Is Promptfoo?
A framework for evaluating and security-testing large language model applications.
An open-source framework for AI evaluation
Promptfoo is an open-source framework for evaluating and security-testing large language model applications. It allows developers to define prompts, supply test inputs, evaluate responses against expected behavior, and compare results in a repeatable workflow.
AI-generated responses can vary depending on how instructions are written, how information is retrieved, and how an application processes user requests. Manual testing can reveal individual problems, but it becomes difficult to compare hundreds of inputs or consistently detect regressions.
Promptfoo addresses this challenge by turning testing into a structured process. Developers can describe their test scenarios in configuration files, run evaluations from a terminal, and review the results using a browser-based interface.
It is useful for developers, AI engineers, security testers, and teams building applications that rely on generated text or automated decisions.
Practical Capabilities for AI Development
Six concrete ways Promptfoo turns AI quality and safety into a measurable, repeatable process.
Evaluate AI responses
Check whether generated outputs satisfy predefined requirements.
Compare prompts
Run the same inputs against different prompt designs and identify behavioral differences.
Test AI security
Examine how an application responds to potentially unsafe or unexpected inputs.
Validate retrieval workflows
Evaluate responses that depend on retrieved documents and supporting context.
Automate testing
Include repeatable evaluations in development and deployment workflows.
Analyze results
Review failures, comparisons, and test outcomes through a visual interface.
Six Pillars of the Promptfoo Workflow
Each capability addresses a specific stage of evaluating and hardening AI-powered applications.
Automated LLM Evaluations
Run structured tests across multiple prompts, datasets, and configurations. Instead of reviewing every output manually, you can define expected conditions and identify which responses satisfy them.
Tests can include exact text checks, content requirements, and more advanced evaluation methods depending on the selected configuration.
AI Red Teaming and Security Testing
AI applications can encounter unexpected instructions, attempts to bypass safeguards, and inputs designed to reveal sensitive information.
Promptfoo supports controlled red teaming to help identify issues such as prompt injection, unsafe responses, information disclosure, and failures to follow application policies.
Security testing should be performed only against systems you own or are authorized to assess.
Prompt Comparison
Small changes in instructions can affect the quality, structure, and consistency of AI responses.
By evaluating multiple prompt versions against the same test cases, developers can examine meaningful differences before selecting an approach for their application.
Prompt A
Returns concise, structured answers. Passes 14 / 16 test cases.
Prompt B
More conversational tone. Passes 12 / 16 test cases.
Custom Test Cases and Assertions
Create tests around the actual behavior your application needs.
You can define inputs, expected outputs, content conditions, and reusable test scenarios. This helps connect evaluation results to real application requirements rather than relying entirely on subjective impressions.
Browser-Based Results Viewer
After completing an evaluation, review the output in a local web interface.
The results viewer provides a convenient way to inspect responses, compare test outcomes, filter failures, and investigate unexpected behavior without reading long terminal logs.
Development Workflow Integration
Promptfoo can be used during development and in automated testing pipelines.
Teams can repeat evaluations after prompt changes, model updates, or application modifications to detect issues before releasing changes to users.
A Repeatable Evaluation Workflow
Promptfoo turns application expectations into testable conditions through four clear stages.
Define Your Test Goal
Start by deciding what you want to measure. This might be response accuracy, formatting, safety, instruction following, or consistency. A clear goal helps you create useful test cases rather than collecting results without a meaningful success condition.
Configure Prompts and Test Inputs
Define the prompts and example inputs that represent the behavior you want to examine. Test cases should include normal requests, difficult edge cases, and situations likely to expose weaknesses.
Run the Evaluation
Execute the configured tests through the command-line interface. Promptfoo sends the specified inputs to the configured provider and checks the results against your assertions.
Review and Improve
Inspect passes, failures, unexpected outputs, and comparison results. Use those findings to revise prompts, improve application logic, and repeat the evaluation. The purpose is not simply to achieve a high test score — it is to build evidence that your AI application behaves as intended under relevant conditions.
Where Teams Apply Promptfoo
Practical scenarios where structured evaluation pays off across the AI development lifecycle.
AI Chatbot Testing
Test whether a chatbot follows instructions, handles difficult requests, returns appropriate answers, and maintains expected behavior.
AI Agent Evaluation
Examine whether agents complete tasks correctly, use tools appropriately, and respect the boundaries of their intended workflow.
Retrieval-Augmented Generation Testing
Evaluate whether answers are supported by retrieved information, remain relevant to the user's question, and avoid unsupported claims.
Prompt Regression Testing
Run familiar test cases after changing prompts or application logic to identify behavior that may have improved or deteriorated.
Automated Development Checks
Include evaluations in a repeatable development process so important changes can be assessed before deployment.
Compatibility & Requirements
Check compatibility before installing the software.
| Requirement | Details |
|---|---|
| Supported operating systems | Windows, macOS, Linux |
| Runtime | Node.js 22.22.0 or newer |
| Recommended runtime | Node.js 24 LTS |
| Installation method | npm or npx |
| Recommended memory | 4 GB RAM or more |
| Recommended storage | 10 GB or more |
| Dedicated GPU | Not required for the CLI |
| Internet connection | Required for installation and many remote-model evaluations |
| Interface | Command line with optional local web viewer |
Get Promptfoo — Free & Open-Source
Promptfoo is primarily distributed as a command-line tool rather than a traditional desktop installer. You can install it globally, run it on demand, or add it to an existing development project.
Installation options
| Method | Command | Best for |
|---|---|---|
| Global installation | npm install -g promptfoo | Regular use |
| On-demand execution | npx promptfoo@latest | First-time users |
| Project dependency | npm install promptfoo --save | Application development |
Before installing: Make sure your computer has Node.js 22.22.0 or newer. Version 24 LTS is recommended.
How to Install Promptfoo
The installation process is similar across supported operating systems, although the terminal application and system configuration may differ.
Check Your Runtime Version
Open a terminal and enter:
node --versionThe installed version must meet the minimum requirement. Next, verify that the package manager is available:
npm --versionIf the runtime is missing or outdated, install a supported version before continuing.
Install Promptfoo
For a global installation, run:
npm install -g promptfooThis makes the command-line tool available for regular use. Alternatively, use an on-demand command without maintaining a permanent global installation:
npx promptfoo@latest --versionVerify the Installation
After global installation, enter:
promptfoo --versionThe terminal should return the installed version number. If the command is not recognized, check whether the installation completed and whether the global package directory is available in your system PATH.
Initialize a Project
Create your first project configuration:
promptfoo initThe setup process helps create the configuration needed to define prompts, providers, and evaluation tests.
Run an Evaluation
Once the configuration is ready, run:
promptfoo evalThe software processes the configured tests and displays their results.
Open the Results Viewer
To inspect the output visually, run:
promptfoo viewA local browser interface lets you examine test cases, compare results, and investigate failures. For a more detailed walkthrough, visit the Promptfoo Installation Guide on this website.
Your First Promptfoo Evaluation Example
Before connecting a real AI model, you can check whether your configuration and variable substitution work correctly with the built-in echo provider.
Create a file named promptfooconfig.yaml containing the prompt, a provider, and a test case as shown alongside. Save the file and run the evaluation command.
You can then open the results viewer to inspect the outcome. In this example, the echo provider returns the rendered input prompt rather than generating a new AI response. This allows you to check basic configuration behavior without an external model account.
Then: npx promptfoo@latest view
prompts: - 'Write a greeting for {{name}}' providers: - echo tests: - vars: name: Alex assert: - type: contains value: Alex
$ npx promptfoo@latest eval ✔ Provider echo returned response ✔ contains:"Alex" — PASS 3 tests, 3 passed, 0 failed $ npx promptfoo@latest view ✔ Viewer ready at http://localhost:15500
AI Red Teaming with Promptfoo
Red teaming is a structured way of examining how an AI application behaves when exposed to adversarial, unexpected, or policy-conflicting inputs. Traditional functional tests usually focus on whether the application performs an intended task. Red teaming instead explores how it might fail.
Prompt Injection Testing
Evaluate whether untrusted instructions can improperly influence application behavior. This is especially relevant when an AI application processes retrieved documents, messages, or other user-controlled content.
Sensitive Information Protection
Check whether an application may expose information that should remain protected. Tests can help identify gaps in application-level protections, although a successful test suite does not prove the complete absence of security vulnerabilities.
Policy and Safety Validation
Evaluate whether an application consistently follows defined behavioral restrictions across relevant scenarios.
Agent and Tool-Use Security
Examine whether AI agents follow authorization boundaries and handle instructions safely when interacting with external tools or application functions. Red teaming is most useful when findings lead to clear remediation steps, improved safeguards, and repeated verification.
Answers to Common Promptfoo Questions
Browse by category or expand any question to read the full answer.
npm install -g promptfoo@latestAfter updating, run
promptfoo --version to confirm the installed version.Common Promptfoo Installation Problems
Quick diagnosis for the issues new users encounter most often.
Command Not Found
If the terminal cannot locate Promptfoo after global installation, verify that the package installed successfully and that the global package directory is included in your PATH. Running the tool through npx may provide an alternative.
Unsupported Runtime Version
An older runtime can prevent installation or cause compatibility issues. Check the active version with node --version and upgrade if necessary.
Configuration File Not Found
Confirm that the configuration file is saved in the project directory and that you're running the evaluation from the correct location. By default, the expected configuration filename is promptfooconfig.yaml.
Authentication Errors
Real model evaluations may require credentials supplied through environment variables or provider configuration. Verify that the appropriate credentials are present without exposing them in shared files or screenshots.
Unexpected Evaluation Results
Review the test input, provider configuration, assertions, and actual response before assuming the software produced an incorrect result. A failed assertion may indicate either a genuine application issue or an overly restrictive test condition.
Explore the Detailed Guides
Develop a stronger understanding of AI testing and application security with these focused resources.
Installation Guide
Set up Promptfoo on your operating system and solve common installation problems.
Configuration Guide
Learn how prompts, providers, variables, and assertions work together.
Evaluation Examples
Explore practical test cases for different application requirements.
Red Teaming Guide
Understand security testing workflows and responsible evaluation practices.
Troubleshooting Guide
Find solutions to configuration errors, runtime issues, and unexpected results.
Start Testing Your AI Applications
A reliable AI application needs more than a working prompt. It needs repeatable tests, measurable expectations, and an effective way to investigate failures. Promptfoo provides a framework for bringing those practices into everyday development.
About this website: This is an independent educational resource focused on installation, tutorials, and software guidance. It is not the official software publisher or an authorized representative. Always verify package integrity and permissions before installing software.