After launch: what it takes to run AI agents in production

Traditional software mostly behaves the same way the day after launch as it did on launch day. AI agents change over time. Customers ask new questions, products and policies change, data drifts and model providers release new versions. An agent left alone gets worse slowly, often without anyone noticing.

That is why we run the agents we build. This is what that work looks like each week.

Monitor the right signals

Uptime and response time still matter, but they say little about whether an agent is doing its job. We watch signals tied to the work:

• Task outcomes: how many requests the agent completes, hands off or abandons
• Quality: scores from automatic graders on a sample of live runs, plus human review
• Hand-off reasons: which kinds of requests the agent cannot handle, and how that changes
• Customer signals: repeat contacts, complaints and ratings after an agent interaction
• Cost per task: so spending stays in line with the value delivered
• Security events: refused actions, injection attempts and unusual access patterns

Each signal has a normal range and an owner. When one moves out of range, someone looks at it the same day.

Review real conversations

Dashboards show that something changed. Reading conversations shows why. Every week we review a sample of runs with someone from the client team, focusing on hand-offs, low scores and anything a customer complained about.

The findings usually fall into a few groups: a policy the agent does not know about yet, a question phrased in a new way, a tool that returns confusing data or an instruction that needs to be clearer. Each finding becomes a fix and a new test case in the evaluation set.

Keep the knowledge current

Many agent errors come from outdated information. Prices change, policies are updated, products are discontinued. We connect the agent to the sources of truth where possible and set up a simple process for everything else: when a policy changes, its owner updates the document, and the agent's knowledge refreshes automatically.

We also track which documents the agent cites most and least. Documents that are never used may be unclear or hard to find; documents behind many wrong answers usually need rewriting.

Handle model updates

Model providers release new versions and retire old ones on their own schedule. A new version may be faster, cheaper or more capable, and it may also behave differently on the cases that matter to a specific business.

We test every model change against the full evaluation set before it reaches production. If the new version is better across the board, we roll it out gradually. If it improves some groups and hurts others, we adjust prompts, keep the old model for those steps or wait. Clients never learn about a model change from a customer complaint.

Improve on a schedule

Running an agent includes making it better. Each month we review the numbers with the client and decide what to improve next: widen the agent's scope to a new type of request, connect another system, cut cost on an expensive step or retire something nobody uses.

Over time this is where most of the value comes from. An agent that handles a modest share of requests at launch can grow steadily as each month's review removes another reason for a hand-off.

What running includes

When we run an agent for a client, the work covers:

• Monitoring with alerts and a named owner for each signal
• Weekly conversation reviews with the client team
• Fixes, with every failure added to the evaluation set
• Knowledge updates connected to the client's sources of truth
• Tested model updates, rolled out gradually
• A monthly review of performance, cost and next improvements

Building an agent takes weeks. Keeping it useful is ongoing work, and it is the part that decides whether the investment pays off.