Let’s prove you can trust your agent in production
1 underlying structure FRAMEWORK*
Test AI agents on the work your business needs done. See what’s ready for production and what needs attention.
*Merriam-Webster
READ ONQualification for production
Your pilot shows promise. Now you need to know which responsibilities the agent can take on, where it needs a person, and what still needs fixing.
Qualification means testing an agent against the requirements of a specific job. Fabrik runs it through realistic situations in simulated business systems, checks its actions and results, and gives your team findings to base a deployment decision on.
How it works
We’re asking AI agents to take on real work. Each dot: work being done today.
Soon, people will depend on that work being done well. Each cube: an agent taking on a task.
Did it refund the right amount? Did it update the right records?
Let’s test one customer refund.
Step 1 of 5, Prepare. Set up a refund test. A customer requests a refund on an order that has already been partly refunded. $500 order · $200 already refunded · $300 remaining. Success: refund the correct amount and update the records..
Step 2 of 5, Run. Let the agent handle the request. The customer asks for $500 back. The agent reads the payment history but still attempts the full refund. Fabrik records each service call. Customer: “I want the full $500 back.” · Order #1082. Read · order and payment history. Attempted refund · $500.
Step 3 of 5, Inspect. See where it went wrong. Only $300 remains, so the service rejects the refund. Fabrik checks both the attempted action and the unchanged balance: no money moved, but the agent made the wrong decision. No records changed. The case is still open.. × Agent requests an allowed amount — failed. ✓ Rejected refund leaves the balance unchanged — passed.
Step 4 of 5, Change. Make a fix and try again. Your team updates the agent and reruns the same test. This time it confirms and refunds $300, updates the order, closes the case, and checks the records. Second run · using your team’s updated agent. ✓ Accepted refund recorded: $300. ✓ Order and case reference the completed payment.
Step 5 of 5, Compare. Compare the two versions. Compare both runs: the original attempted too much; the update completed the allowed refund. Run the remaining tests to check for regressions. Same setup and test case · two versions of the agent. Original version · tried to refund too much. Updated version · completed the allowed refund.
Know what your agent can handle before you put it to work.
Environment ready
Case #184 · Customer 412
OpenOrder #1082 · Customer 412
Partially refundedPayment #706 · Order #1082
$300 remaining01 / 05 — Prepare
A customer requests a refund on an order that has already been partly refunded.
$500 order · $200 already refunded · $300 remaining
The world
We’re asking AI agents to take on real work.
Each dot: work being done today.
Soon, people will depend on that work being done well.
Each cube: an agent taking on a task.
Did it refund the right amount?
Did it update the right records?
Let’s test one customer refund.
CRMOpen
OrdersPartially refunded
Payments$300 remaining
01 / 05 — Prepare
A customer requests a refund on an order that has already been partly refunded.
$500 order · $200 already refunded · $300 remaining
CRMOpen
OrdersPartially refunded
Payments$300 remaining
02 / 05 — Run
The customer asks for $500 back. The agent reads the payment history but still attempts the full refund. Fabrik records each service call.
Customer: “I want the full $500 back.” · Order #1082
CRMOpen
OrdersPartially refunded
Payments$300 remaining
03 / 05 — Inspect
Only $300 remains, so the service rejects the refund. Fabrik checks both the attempted action and the unchanged balance: no money moved, but the agent made the wrong decision.
No records changed. The case is still open.
CRMResolved · payment #706
OrdersFully refunded
Payments$0 remaining
04 / 05 — Change
Your team updates the agent and reruns the same test. This time it confirms and refunds $300, updates the order, closes the case, and checks the records.
Second run · using your team’s updated agent
CRMResolved · payment #706
OrdersFully refunded
Payments$0 remaining
05 / 05 — Compare
Compare both runs: the original attempted too much; the update completed the allowed refund. Run the remaining tests to check for regressions.
Same setup and test case · two versions of the agent
Know what your agent can handle before you put it to work.
How qualification works
Fabrik builds a test environment around your agent, with simulated users, services, and records. Your team reviews the business rules and test cases, then runs the agent through everyday requests and the failures that matter to your workflow.
Business owners define what must go right. Engineers see where the agent falls short and can test the fixes. Everyone works from the same results when deciding whether to release, keep improving, or limit the agent’s responsibilities.
The simulated systems use related customer, order, and payment records. Changes persist through the task, so you can check whether the agent completed the work correctly, including what happened after its first action.
A customer leaves out a detail, records disagree, or a service fails midway. Test whether the agent asks for clarification, follows the required approval process, or hands the task to someone who can help.
After each run, review the request, tool calls, record changes, and failed checks together. Actions and results are checked separately. Engineers can trace a failure, make a change, and test whether it worked.
Beyond the first qualification
Once an agent is in use, its prompts, tools, and business requirements will change. Reuse the test environment to compare versions, check that fixes worked, and catch new failures before the next release.
| Test case | Before → After | Result |
|---|---|---|
| Partial refund | Failed → Passed | Fixed |
| Service timeout | Passed → Failed | New failure |
| Missing customer | Passed → Passed | Unchanged |
Illustrative results showing a fix, a new failure, and an unchanged result.
Your team keeps the environment and test cases from the first qualification, saving the work of setting them up again. Add cases as you learn, including tests based on real incidents. Each release builds on what you already know about the agent.
Compare the same scenarios across two versions to see which failures were fixed, which appeared, and what stayed the same. A better overall result can still hide a new problem in an important task. Your team can review those differences before approving a release.
Your first qualification
Choose an agent, the job you want it to do, and a decision you need to make. That might be putting a pilot into production, expanding its responsibilities, or deciding whether an agent from a vendor meets your needs.
Together, we define what success looks like, which systems the test needs to represent, and which failures matter most. That gives the first qualification a practical scope and a clear purpose.
Your engineers keep the application they’ve built and connect it to the test environment. Agents from vendors can also be tested where suitable access is available. We establish what can be tested and observed as part of setup.
Fabrik engineers can help prepare the environment and run the initial qualification. Your team gets the findings, reviewed test cases, and a starting set of results to compare future versions against.
Setup is scoped around your workflow, available access, and deployment needs. Your team reviews the test assumptions and keeps responsibility for the release decision.
A view of where the agent meets your requirements, where it fails, and what needs more testing. You also get a reusable test environment, reviewed scenarios, and a baseline for future versions. The findings help business owners and engineers decide what to release and what to work on next.
No. Fabrik works with the agent you already have. Setup is lightweight, and our team can help you get connected and ready for qualification.
No. We model the services, data, and behavior needed for the task you choose. Your team reviews the business rules and assumptions. We agree on any specialized behavior or custom integrations during setup.
Yes, if the vendor offers a supported test interface. When the agent can use test services, Fabrik can check its actions and the records it changes. If only a conversation is available, that is what Fabrik can test. The results explain what could and could not be observed.
You can start with synthetic records and simulated services. Existing conversations or activity logs can help make tests more representative, after review for sensitive information. During setup, your team confirms which connections are simulated and which are intentionally live; isolation depends on how the agent is connected.
No. Results apply to the work and conditions you tested. Your team needs to consider how well those tests reflect the real task, what could be observed, and what remains untested. Qualification supports a decision about specific responsibilities; it does not remove the need for judgment or human oversight.
Not always. Fabrik saves the setup, test case, simulated customer, and random seed so you can recreate the starting conditions. Language models and any permitted external services can still behave differently. Repeating a test helps you see that variation.
Yes. Our team can help you set up testing in your CI/CD pipeline. Your team chooses the release rules, such as blocking a version with new critical failures or requiring review when results get worse. Your pipeline enforces those rules.
Have an agent you want to put to work?
Bring one workflow and the questions
you need answered before it goes live.
We’re working with a small group of early partners. Get in touch: founders@fabriklabs.ai
