Promptfoo Promptfoo Versions Save

Test your prompts, agents, and RAGs. Use LLM evals to improve your app's quality and catch problems. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

0.66.0

1 week ago

What's Changed

Features

config get/set commands, ability for users to set their email by @typpo in #971
webui: Download as CSV by @typpo in #1000
Add support for Gemini default grader if credentials are present by @typpo in #998
redteam: Allow arbitrary providers by @mldangelo in #1002

Fixes

Derived metrics by @typpo in #985
Python provider can import modules with same name as built-ins by @typpo in #989
Include error text in all cases by @typpo in #990
Ensure tests inside scenarios are filtered by filter patterns by @mldangelo in #996
Anthropic message API support for env vars by @typpo in #997

Documentation

Add build documentation workflow and fix typos by @mldangelo in #993

Chores

Block network calls in tests by @typpo in #972
Export AnthropicMessagesProvider from providers by @greysteil in #975
Add Claude 3.5 sonnet pricing by @typpo in #976
Pass tool_choice to Anthropic when set in config by @greysteil in #977
Fixed according to Ollama API specifications by @keishidev in #981
Add Dependabot config and update provider dependencies by @mldangelo in #984
Don't commit .env to Git by @will-holley in #991
Update Docker base image to Node 20, improve self-hosting documentation, and add CI action for Docker build by @mldangelo in #995
Allow variable cells to scroll instead of exploding the table height by @grrowl in #973

New Contributors

@keishidev made their first contribution in #981
@grrowl made their first contribution in #973

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.65.2...0.66.0

0.65.2

2 weeks ago

What's Changed

feat(redteam): make it easier to add non default plugins by @typpo in https://github.com/promptfoo/promptfoo/pull/958
feat: contains-sql assert by @typpo in https://github.com/promptfoo/promptfoo/pull/964
fix: handle absolute paths for js providers by @typpo in https://github.com/promptfoo/promptfoo/pull/966
fix(redteam): wire ui to plugins by @typpo in https://github.com/promptfoo/promptfoo/pull/965
fix(redteam): reduce overreliance, excessive-agency false positive rates by @typpo in https://github.com/promptfoo/promptfoo/pull/963
fix: label not showing problem when using eval with config option by @suhwan-kang in https://github.com/promptfoo/promptfoo/pull/928
fix: should return the whole message if the OpenAI return the content and the function call/tools at the same time. by @qwlong in https://github.com/promptfoo/promptfoo/pull/968
fix: label support for js prompts by @typpo in https://github.com/promptfoo/promptfoo/pull/970
docs: Add CLI delete command to docs by @efung in https://github.com/promptfoo/promptfoo/pull/959
docs: text to sql validation guide by @typpo in https://github.com/promptfoo/promptfoo/pull/962

New Contributors

@suhwan-kang made their first contribution in https://github.com/promptfoo/promptfoo/pull/928
@qwlong made their first contribution in https://github.com/promptfoo/promptfoo/pull/968

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.65.1...0.65.2

0.65.1

2 weeks ago

Fixes related to web UI. See also 0.65.0 release notes

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.65.0...0.65.1

0.65.0

2 weeks ago

What's Changed

feat(webui): show pass/fail toggle by @typpo in https://github.com/promptfoo/promptfoo/pull/938
feat(webui): carousel for multiple failure reasons by @typpo in https://github.com/promptfoo/promptfoo/pull/939
feat(webui): clicking metric pills filters by nonzero only by @typpo in https://github.com/promptfoo/promptfoo/pull/941
feat: add hf to onboarding flow by @typpo in https://github.com/promptfoo/promptfoo/pull/947
feat: add support for promptfoo export latest by @typpo in https://github.com/promptfoo/promptfoo/pull/948
feat(redteam): political statements by @typpo in https://github.com/promptfoo/promptfoo/pull/944
feat(redteam): indicate performance with moderation filter by @typpo in https://github.com/promptfoo/promptfoo/pull/933
fix: serialize each item in vars when its type is a string (#823) by @YiFanChen99 in https://github.com/promptfoo/promptfoo/pull/943
fix(redteam): more aggressive contract testing by @typpo in https://github.com/promptfoo/promptfoo/pull/946
chore(deps): update dependencies without breaking changes by @mldangelo in https://github.com/promptfoo/promptfoo/pull/937
chore(webui): split ResultsTable into separate files by @typpo in https://github.com/promptfoo/promptfoo/pull/942

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.64.0...0.65.0

0.64.0

2 weeks ago

What's Changed

feat: export promptfoo.cache to node package by @typpo in https://github.com/promptfoo/promptfoo/pull/923
feat: add Voyage AI embeddings provider by @typpo in https://github.com/promptfoo/promptfoo/pull/931
feat(redteam): add unintended contracts test by @mldangelo in https://github.com/promptfoo/promptfoo/pull/934
feat(anthropic): support tool use by @mldangelo in https://github.com/promptfoo/promptfoo/pull/932
feat: Add more Portkey header provider options and create headers automatically by @vrushankportkey in https://github.com/promptfoo/promptfoo/pull/909
fix: handle openai chat-style messages better in moderation assert by @typpo in https://github.com/promptfoo/promptfoo/pull/930
fix(bedrock): support cohere embeddings by @typpo in https://github.com/promptfoo/promptfoo/pull/924
ci: add next.js build caching by @typpo in https://github.com/promptfoo/promptfoo/pull/908
chore(deps): bump braces from 3.0.2 to 3.0.3 by @dependabot in https://github.com/promptfoo/promptfoo/pull/918
chore(docs): update installation and GitHub Actions guides by @mldangelo in https://github.com/promptfoo/promptfoo/pull/935
chore(dependencies): bump LLM providers in package.json by @mldangelo in https://github.com/promptfoo/promptfoo/pull/936

New Contributors

@vrushankportkey made their first contribution in https://github.com/promptfoo/promptfoo/pull/909

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.63.2...0.64.0

0.63.2

3 weeks ago

What's Changed

fix(bedrock): default value for configs by @typpo in https://github.com/promptfoo/promptfoo/pull/917

0.63.1

3 weeks ago

What's Changed

fix(vertex): correct handling of system instruction by @typpo in https://github.com/promptfoo/promptfoo/pull/911
fix(bedrock): support for llama, cohere command and command-r, mistral by @typpo in https://github.com/promptfoo/promptfoo/pull/915

See also 0.63.0 release notes

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.63.0...0.63.1

0.63.0

3 weeks ago

What's Changed

feat(bedrock): Add support for mistral, llama, cohere by @typpo in https://github.com/promptfoo/promptfoo/pull/885
feat(ollama): add OLLAMA_API_KEY to support authentication by @syndimann in https://github.com/promptfoo/promptfoo/pull/883
feat(redteam): add test for competitor recommendations by @typpo in https://github.com/promptfoo/promptfoo/pull/877
feat(webui): Show the number of passes and failures by @typpo in https://github.com/promptfoo/promptfoo/pull/888
feat(webui): show manual grading record in test details view by @typpo in https://github.com/promptfoo/promptfoo/pull/906
feat(webui): use indexeddb instead of localstorage by @typpo in https://github.com/promptfoo/promptfoo/pull/905
feat: ability to set test case metric from csv by @typpo in https://github.com/promptfoo/promptfoo/pull/889
feat: interactive onboarding by @typpo in https://github.com/promptfoo/promptfoo/pull/886
feat: support threshold param from csv by @typpo in https://github.com/promptfoo/promptfoo/pull/903
feat: support array of values for similar assertion by @typpo in https://github.com/promptfoo/promptfoo/pull/895
fix(vertex): support var templating in system instruction by @typpo in https://github.com/promptfoo/promptfoo/pull/902
fix: Prompt variable reads unprocessed spaces on both sides by @big-camel in https://github.com/promptfoo/promptfoo/pull/887
fix: windows node 22 flake by @typpo in https://github.com/promptfoo/promptfoo/pull/907
fix(webui): display latency when available
fix: ci passing despite failing build (https://github.com/promptfoo/promptfoo/pull/876)
fix: incorrect migrations path in docker build https://github.com/promptfoo/promptfoo/issues/861
chore(ci): add workflow_dispatch trigger by @mldangelo in https://github.com/promptfoo/promptfoo/pull/897
chore(deps): update most dependencies to latest stable versions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/898
chore: add more gemini models by @typpo in https://github.com/promptfoo/promptfoo/pull/894
chore: introduce eslint by @mldangelo in https://github.com/promptfoo/promptfoo/pull/904
chore: switch to SWC for faster Jest tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/899
chore: update to prettier 3 by @mldangelo in https://github.com/promptfoo/promptfoo/pull/901
chore(openai): add tool_choice required type

New Contributors

@syndimann made their first contribution in https://github.com/promptfoo/promptfoo/pull/883

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.62.1...0.63.0

0.62.1

3 weeks ago

What's Changed

fix: migrate database before writing results by @typpo in https://github.com/promptfoo/promptfoo/pull/882
feat(webui): Ability to suppress browser open on promptfoo view by @typpo in https://github.com/promptfoo/promptfoo/pull/881

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.62.0...0.62.1

Notes from 0.62.0 below

What's Changed

chore: upgrade default graders to gpt-4o by @typpo in https://github.com/promptfoo/promptfoo/pull/848
ci: Introduce jest test coverage reports by @mldangelo in https://github.com/promptfoo/promptfoo/pull/868
ci: add support for node 22, remove support for node 16 by @jacobdr in https://github.com/promptfoo/promptfoo/pull/836
docs: Addresses minor typographical errors by @mldangelo in https://github.com/promptfoo/promptfoo/pull/845
docs: Help description of default --output by @jamesbraza in https://github.com/promptfoo/promptfoo/pull/844
feat(anthropic): add support for base url by @typpo in https://github.com/promptfoo/promptfoo/pull/850
feat(openai): Support function/tool callbacks by @typpo in https://github.com/promptfoo/promptfoo/pull/830
feat(vertex/gemini): add support for toolConfig and systemInstruction by @typpo in https://github.com/promptfoo/promptfoo/pull/841
feat(webui): Ability to filter to highlighted cells by @typpo in https://github.com/promptfoo/promptfoo/pull/852
feat(webui): ability to click to filter metric by @typpo in https://github.com/promptfoo/promptfoo/pull/849
feat(webui): add copy and highlight cell actions by @typpo in https://github.com/promptfoo/promptfoo/pull/847
feat: Add Red Team PII Tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/862
feat: Support custom gateway URLs in Portkey by @roh26it in https://github.com/promptfoo/promptfoo/pull/840
feat: add support for python embedding and classification providers by @typpo in https://github.com/promptfoo/promptfoo/pull/864
feat: add support for titan premier on bedrock by @YiFanChen99 in https://github.com/promptfoo/promptfoo/pull/839
feat: pass evalId in results by @pelikhan in https://github.com/promptfoo/promptfoo/pull/758
fix: Broken types by @brendanmoore in https://github.com/promptfoo/promptfoo/pull/854
fix: Fix broken progress callback in web ui by @chrismaltais in https://github.com/promptfoo/promptfoo/pull/860
fix: Fix formatting and add style check to CI by @chrismaltais in https://github.com/promptfoo/promptfoo/pull/872
fix: Fix type error eval page.tsx by @drien in https://github.com/promptfoo/promptfoo/pull/867
fix: Improve Error Handling for Python Assertions and Provider Exceptions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/863
fix: Pass evaluateOptions from web ui yaml by @chrismaltais in https://github.com/promptfoo/promptfoo/pull/859
fix: Render multiple result images with markdown, if markdown contains multiple images by @chrismaltais in https://github.com/promptfoo/promptfoo/pull/873
fix: The values of defaultTest and evaluateOptions are not set when editing the eval yaml file. by @big-camel in https://github.com/promptfoo/promptfoo/pull/834
fix: crash on db migration when cache is disabled on first run by @typpo in https://github.com/promptfoo/promptfoo/pull/842
fix: csv and html outputs include both prompt and provider labels by @typpo in https://github.com/promptfoo/promptfoo/pull/851
fix: docker build and prepublish script by @typpo in https://github.com/promptfoo/promptfoo/pull/846
fix: show labels for custom provider by @typpo in https://github.com/promptfoo/promptfoo/pull/875

New Contributors

@YiFanChen99 made their first contribution in https://github.com/promptfoo/promptfoo/pull/839
@roh26it made their first contribution in https://github.com/promptfoo/promptfoo/pull/840
@pelikhan made their first contribution in https://github.com/promptfoo/promptfoo/pull/758
@brendanmoore made their first contribution in https://github.com/promptfoo/promptfoo/pull/854
@drien made their first contribution in https://github.com/promptfoo/promptfoo/pull/867

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.61.0...0.62.0

0.62.0

4 weeks ago

What's Changed

feat(anthropic): add support for base url by @typpo in https://github.com/promptfoo/promptfoo/pull/850
feat(openai): Support function/tool callbacks by @typpo in https://github.com/promptfoo/promptfoo/pull/830
feat(vertex/gemini): add support for toolConfig and systemInstruction by @typpo in https://github.com/promptfoo/promptfoo/pull/841
feat(webui): Ability to filter to highlighted cells by @typpo in https://github.com/promptfoo/promptfoo/pull/852
feat(webui): ability to click to filter metric by @typpo in https://github.com/promptfoo/promptfoo/pull/849
feat(webui): add copy and highlight cell actions by @typpo in https://github.com/promptfoo/promptfoo/pull/847
feat: Add Red Team PII Tests by @mldangelo in https://github.com/promptfoo/promptfoo/pull/862
feat: Support custom gateway URLs in Portkey by @roh26it in https://github.com/promptfoo/promptfoo/pull/840
feat: add support for python embedding and classification providers by @typpo in https://github.com/promptfoo/promptfoo/pull/864
feat: add support for titan premier on bedrock by @YiFanChen99 in https://github.com/promptfoo/promptfoo/pull/839
feat: pass evalId in results by @pelikhan in https://github.com/promptfoo/promptfoo/pull/758
feat: upgrade default graders to gpt-4o by @typpo in https://github.com/promptfoo/promptfoo/pull/848
fix: Broken types by @brendanmoore in https://github.com/promptfoo/promptfoo/pull/854
fix: Fix broken progress callback in web ui by @chrismaltais in https://github.com/promptfoo/promptfoo/pull/860
fix: Fix formatting and add style check to CI by @chrismaltais in https://github.com/promptfoo/promptfoo/pull/872
fix: Fix type error eval page.tsx by @drien in https://github.com/promptfoo/promptfoo/pull/867
fix: Improve Error Handling for Python Assertions and Provider Exceptions by @mldangelo in https://github.com/promptfoo/promptfoo/pull/863
fix: Pass evaluateOptions from web ui yaml by @chrismaltais in https://github.com/promptfoo/promptfoo/pull/859
fix: Render multiple result images with markdown, if markdown contains multiple images by @chrismaltais in https://github.com/promptfoo/promptfoo/pull/873
fix: The values of defaultTest and evaluateOptions are not set when editing the eval yaml file. by @big-camel in https://github.com/promptfoo/promptfoo/pull/834
fix: crash on db migration when cache is disabled on first run by @typpo in https://github.com/promptfoo/promptfoo/pull/842
fix: csv and html outputs include both prompt and provider labels by @typpo in https://github.com/promptfoo/promptfoo/pull/851
fix: docker build and prepublish script by @typpo in https://github.com/promptfoo/promptfoo/pull/846
fix: show labels for custom provider by @typpo in https://github.com/promptfoo/promptfoo/pull/875
ci: Introduce jest test coverage reports by @mldangelo in https://github.com/promptfoo/promptfoo/pull/868
ci: add support for node 22, remove support for node 16 by @jacobdr in https://github.com/promptfoo/promptfoo/pull/836
docs: Addresses minor typographical errors by @mldangelo in https://github.com/promptfoo/promptfoo/pull/845
docs: Help description of default --output by @jamesbraza in https://github.com/promptfoo/promptfoo/pull/844

New Contributors

@YiFanChen99 made their first contribution in https://github.com/promptfoo/promptfoo/pull/839
@roh26it made their first contribution in https://github.com/promptfoo/promptfoo/pull/840
@pelikhan made their first contribution in https://github.com/promptfoo/promptfoo/pull/758
@brendanmoore made their first contribution in https://github.com/promptfoo/promptfoo/pull/854
@drien made their first contribution in https://github.com/promptfoo/promptfoo/pull/867

Full Changelog: https://github.com/promptfoo/promptfoo/compare/0.61.0...0.62.0