Frequently asked questions
Is EvalDog free?
Yes — the dashboard and the CLI are free to use. A Pro plan ($29/mo) with scheduled runs and model-drift alerts is coming.
Does it cost LLM tokens to run?
No. EvalDog grades the outputs you already have against assertions (contains, equals, regex, is-json, not-empty) — deterministic, zero model calls. Run it 1,000× for $0 in tokens.
What file formats does it take?
CSV, JSON, or YAML. CSV columns: name, output, expected, assert. It also reads promptfoo-style { tests: [...] } files.
Do I need to sign in to try it?
No. Open the dashboard, run a sample, and see a graded report — no account needed to start.
How is my data handled?
Your test cases are processed only to produce your report. We never train on your data. If you add provider API keys for model-graded evals, they are encrypted and used solely to run your evaluations. See our Privacy Policy.
Can I use it in CI / with an AI agent?
Yes — that is the point of the CLI. npx evaldog run cases.csv --min 80 --json exits non-zero on failure, so an agent or pipeline gets a pass/fail without burning context.
How is this different from free Promptfoo?
Same idea, same kind of engine — but hosted, with a dashboard, run history, and (soon) model-drift alerts that re-grade when a provider ships a new model. You skip the self-hosting and the cron.
Who is it for?
QA engineers testing AI features, indie devs shipping LLM apps, and small teams who want a quality gate without standing up an ML platform.
Still stuck? Contact us or read the quick start.