ToolGauntlet
Deterministic chaos testing for tool-using AI agents.
If an agent can’t handle failure in staging, it won’t survive production.
Install the open package, run the included deterministic quickstart without an API key, then add a reliability gate to the workflow that matters.
pip install toolgauntletSelf-serve core available now. Paid implementation pilots start at $2,500.
Run the included deterministic quickstart
Generate a working example, run a built-in suite, and produce a report before connecting a real agent workflow.
pip install toolgauntlet
toolgauntlet quickstart --out-dir toolgauntlet-quickstart
toolgauntlet run --suite ecommerce_refunds_v1 --runs 10 --out report.json
toolgauntlet report --in report.json --format html --out report.htmlThe package, primary CLI, and public Python import are all toolgauntlet.
Test the failures demos miss
ToolGauntlet runs tool-using agents through realistic multi-step workflow tests under controlled failure conditions. Every run is reproducible, so teams can replay failures, debug them quickly, and verify fixes with confidence.
What it tests
- tool timeouts and retry storms
- schema drift and partial payloads
- contradictory tool responses
- unsafe trust in tool-provided text
- regression against previous reliability baselines
- optional OpenAI-compatible BudgetGuard proxy controls for budget, rate, and safety policy
Why it matters
Most agent failures do not come from simple Q&A quality. They happen inside tool loops: retries spiral, schemas drift, APIs return partial data, or an agent trusts the wrong tool output at the wrong time. These failures are expensive, hard to reproduce, and usually discovered too late.
ToolGauntlet is built to make those failure modes testable.
With ToolGauntlet, teams can
- standardize reliability gates across agent workflows
- catch regressions before merge or deployment
- reproduce intermittent failures with fixed seeds
- understand why an agent failed, not just that it failed
- add budget, rate, and policy controls around model usage
Who it’s for
- platform teams building shared agent infrastructure
- QA and automation teams responsible for release quality
- product teams shipping agents that depend on tools and external APIs
Self-serve core, paid implementation pilot
Use the package on your own, or bring HiLo Labs one representative workflow. A paid pilot includes one workflow suite, one baseline report, one CI release gate, and a review of the failure modes uncovered.
Starting at $2,500. The package, primary CLI, and public Python import are all toolgauntlet.
Pilot scope includes:
- packaged built-in suites for common workflow categories
- CI-ready reporting and regression controls
- signed suite packs for controlled distribution
- policy-aware proxy deployment for budget and governance controls
- documentation, examples, and launch materials for early adopters
What a pilot produces
The first milestone is a working reliability gate your team can rerun: one representative workflow, a fixed-seed failure suite, a baseline score, and a report that explains the regressions worth fixing.
Buyer-facing output
- baseline scorecard for a real agent workflow
- replayable failures with seed, injector, and step details
- Markdown or HTML report for engineering review
- CI threshold that can block a risky release
- follow-up recommendations for retry, schema, policy, and budget controls
suite: support_triage_v1
seed: 20260514
baseline reliability: 0.91
candidate reliability: 0.78
release gate: fail
regressions:
- timeout recovery dropped 18%
- partial payload handling dropped 11%
- unsafe tool-text trust observed in 2 runsToolGauntlet Reliability Pilot
Package this into a focused reliability engagement: one representative agent workflow, a fixed-seed chaos suite, a baseline scorecard, and a CI-ready release gate recommendation.
Talk about a ToolGauntlet reliability pilot
Work email is enough to start. Add the workflow you want to test if it is easy to describe now.