Your AI support agent resolved 14,000 tickets last month. Your deflection rate is 73%. Your CEO is happy. But customers are still emailing the human team about the same three problems. Deflection measures cost. It does not measure whether the customer actually got help. That gap is exactly what [product analytics for AI agents](/blog/what-is-locus) was built to close.
Deflection rate is the metric every AI support team reports first. It counts the percentage of inbound tickets the AI agent handled without a human touching them. It is useful for finance. It tells the CFO how much headcount the agent displaced. It tells the product team almost nothing. A ticket can be deflected and still leave the customer confused. A ticket can be deflected and trigger a re-contact two days later. A ticket can be deflected and push the customer toward a competitor's help forum instead. The number stays green while value erodes. This is the same structural problem that appears across all AI agents: why agents pass evals but still fail users applies to support just as much as it applies to coding or writing agents.
Why is deflection rate not enough to measure AI support agent success?
Deflection rate is a containment metric. It answers one question: did the customer leave without escalating? It does not answer: did the customer get the answer they needed? These are different questions. In traditional call centers, they were roughly the same because a human agent could read the room. In AI support, they diverge sharply.
Here is why. An AI support agent can respond confidently with the wrong answer. The customer reads it, assumes the brand is correct, and moves on. The ticket closes. The deflection rate goes up. Two days later the customer discovers the answer was wrong and contacts support again. That re-contact might get handled by a different agent instance, with no memory of the first interaction. The system counts two successful deflections. The customer experienced one failure.
In a study of AI chatbot interactions across e-commerce support, 22% of deflected tickets resulted in a re-contact within 72 hours about the same issue (illustrative, based on Locus snapshot data). That means roughly one in five closed tickets was not actually resolved. The deflection metric missed all of them.
What metrics should you track for an AI customer support agent?
Five metrics together give you a real picture of whether your AI support agent is creating value for customers. No single metric is sufficient. Each one catches a failure mode the others miss. They should be tracked per intent cluster, not in aggregate, because an agent that excels at password resets may fail badly at billing disputes.
1. Resolution confidence.
Resolution confidence is an estimate of whether the customer's stated problem was actually solved by the agent's response. It is not binary. It is a score derived from the conversation's trajectory: did the customer confirm the fix? Did they ask a follow-up that implies confusion? Did they go silent after a complex answer? A resolution confidence below 0.6 on a 0-to-1 scale for any intent cluster is a signal the agent is producing answers, not solutions (illustrative). Tools like Langfuse and Braintrust log the conversation. Locus reads the trajectory and scores the confidence.
2. Re-contact rate.
Re-contact rate is the percentage of closed tickets where the same customer contacts support again about the same issue within a defined window, typically 72 hours. A healthy AI support agent keeps re-contact rate below 12% (illustrative). Above that, the agent is closing tickets without resolving problems. Re-contact rate is the behavioral equivalent of shadow rework for support: the system says success, the customer says failure.
3. Escalation quality.
Not every escalation is a failure. A well-designed support agent should escalate when it recognizes it cannot help. Escalation quality measures whether the handoff to a human is useful: does the human agent have enough context to pick up without asking the customer to repeat everything? In production, 41% of AI-to-human escalations result in the human asking the customer to re-explain the issue (illustrative). That means the escalation path exists but the context transfer is broken.
4. Time-to-resolution vs time-to-re-contact.
Time-to-resolution measures how quickly the agent closes the ticket. Time-to-re-contact measures how quickly the customer comes back about the same issue. The ratio between them tells you whether speed came at the cost of quality. An agent that resolves in 45 seconds but triggers re-contacts in 48 hours is optimizing for the wrong metric. The pattern is the same one that makes trust in AI agent outputs so hard to see: the speed metric looks good while the value metric degrades quietly.
5. Downstream action rate.
Downstream action rate measures whether the customer did the thing the agent told them to do. If the agent says "go to Settings > Billing > Update card," downstream action rate checks whether the customer actually visited that page within the session. If the agent says "your refund will appear in 3-5 business days," it checks whether the customer filed a dispute anyway. This is the support-specific version of time-to-trust-action. A downstream action rate below 40% means the agent is giving instructions customers do not follow (illustrative). Either the instructions are wrong, unclear, or the customer does not believe them.
Why should you measure AI support metrics per intent, not in aggregate?
An AI support agent handles dozens of different request types. Password resets are structurally different from billing disputes. How-to questions are different from bug reports. An aggregate deflection rate of 73% might mean the agent is 95% effective on password resets and 31% effective on billing disputes. The aggregate hides the disaster.
Intent-level measurement breaks the total volume into clusters of similar requests and scores each cluster independently. This is the same logic behind behavioural user segmentation for product agents: the average user does not exist. The average support request does not exist either. What exists is a distribution of intents, each with its own success profile.
In a typical Locus snapshot of a support agent, we find 6 to 12 distinct intent clusters accounting for 80% of volume (illustrative). The top 3 clusters usually perform well because they are the ones the agent was trained on most heavily. Clusters 4 through 8 are where the failures hide. They represent real customer needs that the agent handles with surface confidence and poor outcomes.
What do existing support tools miss about AI agent measurement?
Traditional support platforms like Zendesk, Intercom, and Freshdesk track ticket-level metrics: first response time, resolution time, CSAT, ticket volume. These metrics were designed for human agents. They assume the agent understood the problem because a human read it. They assume resolution means value because a human confirmed it. Neither assumption holds for AI agents.
Observability tools like Datadog and OpenTelemetry tell you the system stayed up. Trace stores like Langfuse and LangSmith show you what the model said. Neither shows you whether the customer got value from what the model said. The measurement gap between system performance and customer value is the same gap that exists across all AI products. The data is already there in conversation logs, re-contact patterns, and downstream events. It just needs a layer that reads them together.
How do you start measuring AI support agent value today?
You do not need to rebuild your analytics stack. Most of the data already exists in three places: your conversation logs (stored in whatever trace store you use), your ticketing system (Zendesk, Intercom, Freshdesk), and your product event stream (Mixpanel, Amplitude, or a data warehouse). The problem is that these three systems do not talk to each other at the conversation level.
- Tag re-contacts. Link tickets from the same customer about the same issue within 72 hours. Most ticketing systems do not do this automatically. Build a simple matcher on customer ID + intent similarity.
- Instrument downstream actions. For your top 3 intent clusters, define what a successful downstream action looks like. Log whether it happens within the session or within 24 hours.
- Score resolution confidence. Use the conversation trajectory, not a binary open/closed flag. Customer confirmation, follow-up questions, and silence patterns all contribute to the score.
- Break metrics by intent cluster. Stop reporting aggregate deflection. Report per-cluster resolution confidence and re-contact rate instead. The CFO still gets the cost number. The product team gets the value number.
- Track weekly. Support intent distributions shift. A metric that looked good last month can degrade as customer questions evolve. Weekly per-intent tracking catches the drift before quarterly reviews do.
If you want to skip the instrumentation work, product analytics for AI agents reads conversation logs from your existing trace store and produces all five metrics per intent cluster in the first memo. The first snapshot is free.
Frequently asked questions.
How do you measure success of an AI customer support agent?
Success of an AI customer support agent is measured through five signals: resolution confidence, re-contact rate, escalation quality, time-to-resolution vs time-to-re-contact ratio, and downstream action rate. Deflection rate alone measures cost savings, not customer value. These five signals, tracked per intent cluster, reveal where the agent creates real value and where it creates the illusion of resolution.
What is a good deflection rate for an AI support agent?
Industry benchmarks range from 40% to 80% depending on the complexity of the product and the support domain. But deflection rate without re-contact rate is misleading. A 73% deflection rate with a 22% re-contact rate means only about 57% of tickets were truly resolved (illustrative). Track both together to understand actual resolution, not just containment.
What is the difference between deflection rate and resolution rate for AI support agents?
Deflection rate counts tickets that closed without a human agent. Resolution rate counts tickets where the customer's problem was actually solved. They overlap but are not the same. A ticket can be deflected without being resolved (the customer gave up). A ticket can be resolved without being deflected (a human stepped in). For AI support measurement, resolution rate per intent cluster is more useful than aggregate deflection.
How do you know if an AI support agent is failing silently?
Silent failure in AI support shows up in three patterns: rising re-contact rates on the same issues, declining downstream action rates (customers not following the agent's instructions), and stable deflection rates paired with dropping CSAT on re-surveyed tickets. The agent closes tickets confidently while customers remain confused. Product analytics for AI agents detects these patterns by reading the full conversation trajectory, not just the close event.
What metrics matter most for AI chatbot KPIs?
The metrics that matter depend on whether you are measuring cost or value. For cost: deflection rate, cost per ticket, and handle time. For value: resolution confidence, re-contact rate, downstream action rate, and per-intent success rate. Most teams report only cost metrics to leadership. Adding value metrics changes what the team builds next.
Can you measure AI support agent quality without CSAT surveys?
Yes. CSAT response rates for AI interactions are typically below 5% (illustrative). Behavioural signals like re-contact rate, downstream action rate, and conversation trajectory analysis cover 100% of interactions. They are also more honest. Customers rate generously in the moment but vote with their re-contacts over time. For the general framework, see how to measure trust in AI agent outputs.