Call-bot.ai

AI agents & agentic automation11 min read

Customer Service AI Agents: Examples, ROI and Risks

A customer service AI agent does more than answer questions: it acts in your order, billing or booking systems and hands the case to a person when it should. The evidence is mixed: one randomized field experiment found faster chats but lower customer ratings on the cases the AI handled, and documented deployments include errors and course corrections. This guide separates sourced results from vendor claims.

Two customer support agents wearing headsets work side by side on laptops at a white desk in a bright office

Photo: MART PRODUCTION · Pexels

In February 2024, Klarna said its AI assistant had handled two-thirds of its customer service chats in its first month, doing the work of 700 full-time agents. In May 2025, PYMNTS reported that Klarna's CEO had told Bloomberg that putting cost first had lowered quality and that the company was undertaking a hiring effort so customers would always have the option of talking to a human. This guide asks what an AI agent for customer service can be trusted with, and how you would find out. We keep three kinds of evidence apart: controlled studies, company-reported figures and documented incidents.

What a customer service AI agent does that a chatbot does not

A chatbot retrieves information from a script or a knowledge base. An AI customer support agent is built to complete the request. Gartner's August 2026 survey commentary describes customers expecting AI to book appointments, place orders, submit documents or manage subscriptions, with the language model handling intent and conventional automation completing the transaction. Klarna, for example, described its assistant as capable of managing refunds and returns.

Three properties set an agent apart: tools (connections to an order database, billing platform or calendar), written policies (a refund limit, an identity check, decisions it may not make) and handoff to a person. Gartner advises attempting resolution only when confidence is high, with a clear path to human support. The research benchmark τ-bench scores agents on whether the database ends in the right state, not on how fluent the conversation sounded.

Treat the label with caution: in June 2025 Gartner estimated that only about 130 of the thousands of agentic AI vendors are real, and said many engage in "agent washing", rebranding chatbots, assistants or robotic process automation without substantial agentic capabilities. For phone calls, see what an AI voice agent is; for the technology, conversational AI.

Deployments: what was claimed, and by whom

Klarna (company-reported, February 27, 2024). A press release said the assistant, powered by OpenAI, had 2.3 million conversations in its first month, two-thirds of Klarna's customer service chats. Klarna reported satisfaction on par with human agents, 25% fewer repeat inquiries and resolution in under 2 minutes instead of 11, and called a $40 million profit improvement for 2024 an estimate. In May 2025, PYMNTS reported that the CEO, speaking to Bloomberg, said cost had been too dominant a factor and had lowered quality, and that Klarna was hiring so customers could reach a human. PYMNTS added that Klarna was not abandoning AI.

Salesforce on its own help site (company-reported; page read October 21, 2026, figures undated). Salesforce says its Agentforce agent has passed five million conversations and resolved more than 63% of them instantly. It defines "resolved" as the agent having answered the customer's inquiry, with abandoned and handed-off conversations counted separately. That is narrower than "the problem was fixed."

What the evidence says about results

A controlled test. A May 2026 arXiv preprint (researchers at Zhejiang University, Fudan University, Dartmouth's Tuck School and Alibaba) reports a randomized field experiment on Taobao in August 2024: 647 workers, 680,676 chats. Treated workers supervised an AI system handling "AI-eligible" chats, under 10% of volume. Chat duration fell 3.2% overall and 16.8% on AI-eligible chats. Repeat contact within seven days did not change significantly. Ratings on AI-eligible chats fell 0.412 points on a five-point scale (only 17% of chats were rated), while ratings across all chats showed no significant change. Caveats: a preprint; Alibaba itself ran the experiment, with an AI system it developed in May 2024; and one author works at Alibaba.

Assisting humans is a different test. In a study of customer support agents given a generative AI assistant (Brynjolfsson, Li and Raymond), issues resolved per hour rose 15% on average in the version published in the Quarterly Journal of Economics in 2025, which reports 5,172 agents. The earlier NBER Working Paper 31161, revised November 2023, reported 5,179 agents, a 14% average gain and 34% for novice and low-skilled workers, with minimal impact on experienced ones; in the published version, the most experienced workers saw small gains in speed and small declines in quality. Check which version a citation uses.

What customers say. Gartner's December 2023 survey of 5,728 customers: 64% would prefer companies not use AI for customer service, and 53% would consider switching to a competitor if they found out a company was going to use it. Its February and March 2026 survey of 3,566 customers: 50% say interactions are easier with generative AI, and 87% say access to a human is essential.

Benchmarks and forecasts. τ-bench (June 2024) found that state-of-the-art agents such as GPT-4o completed under half its tasks and were inconsistent across repeated trials (pass^8 under 25% in retail). Models have changed since; treat it as a reason to test consistency. Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls. The release covers agentic AI projects in general, not customer service specifically, and does not say how the figure was derived.

ROI. None of the sources we read measures net financial return against a control group; the financial figures are company estimates or analyst predictions. For cost lines beyond unit prices, see the hidden costs of an AI voice agent.

Documented failures

Air Canada (decision of February 14, 2024). Jake Moffatt asked the airline's website chatbot about bereavement fares. As reported by Law360 Canada, the chatbot said a reduced rate could be requested within 90 days of ticket issue, while a linked page said the policy did not apply after travel. Air Canada later told Moffatt that retroactive applications were not permitted and, as the tribunal put it, effectively argued that the chatbot was a separate legal entity responsible for its own actions. In Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia's Civil Resolution Tribunal rejected that: the chatbot was part of the website, and the airline was responsible for all information on it, from a static page or a chatbot. It found negligent misrepresentation, because Air Canada had not taken reasonable care to ensure the chatbot was accurate, and held Moffatt's reliance on it reasonable. The award was 650.88 Canadian dollars in damages plus interest and tribunal fees, 812.02 in total. It is one tribunal's decision on its facts, and it does not say how the chatbot worked (Air Canada gave the tribunal no information on that); this summary is not legal advice.

Cursor (April 2025). Its AI support assistant told a user the product allowed one device per subscription. No such policy existed, and a co-founder wrote that the answer was incorrect. eWeek reported refunds for affected users and labeling of AI-generated support replies.

A regulator's warning (CFPB, June 2023). The U.S. Consumer Financial Protection Bureau described "doom loops" in consumer-finance chatbots (customers cycling through unhelpful answers with no route to a person), a limited ability to recognize disputes, and language-model bots that sometimes give inaccurate information. It stated that chatbots must comply with all applicable federal consumer financial laws.

Guardrails that answer these failures

  • Scope limits. List the intents the agent may handle and the actions it may take. Refund exceptions, policy interpretations and disputes go to a person or need approval.
  • One approved source. Air Canada's chatbot and its web page disagreed; Cursor's bot invented a rule. Keep one dated knowledge base, retire contradicting pages, and have the agent escalate instead of improvising.
  • Early escalation. In the Alibaba experiment, human takeovers worked less well once customers were already frustrated. Escalate on a request for a person, repeated failure and frustration signals.
  • Disclosure, permissions, logging and shutdown. Tell customers they are talking to an AI. The rest is in our AI agent governance and security checklist.

Metrics that matter, and how to measure them

Read the vendor's definition of "resolved" first. Intercom's help center, which lists $0.99 per resolution for its Fin agent on chat and email used with Intercom (read October 21, 2026; Intercom's Fin pricing page says billing is in US dollars only; voice and high-volume pricing go through Intercom's sales team; and seat fees come on top), counts a resolution when the customer confirms the answer or leaves without asking for more help, and does not bill when a customer explicitly asks for a person. Salesforce counts an answered inquiry. Neither equals a solved problem. Track:

  1. True resolution: no repeat contact on the same issue within a window you set (the Alibaba study used seven days), checked on an audited sample.
  2. Handoff rate and reasons, and the wait to reach a person.
  3. Accuracy: a person scores a weekly random sample against the approved policy, counting wrong statements and wrong actions.
  4. Ratings split by AI-handled and human-handled contacts, since speed and ratings can diverge.
  5. Consistency: rerun the same scenarios several times, as τ-bench does.
  6. Fully loaded cost per resolved request: platform fees, telephony, integration, quality review, human time on handoffs and the cost of errors, divided by truly resolved requests.

We give no target values: Salesforce and Intercom, for example, count "resolved" in different ways, so your pre-launch baseline is the comparison. For how we compare voice agents, see our methodology.

An implementation sequence

  1. Measure a baseline: volume by intent, handling time, repeat contact, ratings, cost per contact.
  2. Pick narrow, frequent, rule-based, reversible intents, such as order status or appointment changes.
  3. Write the policy and one approved knowledge source, including decisions the agent cannot make and handoff rules.
  4. Connect tools with minimum permissions, read-only first.
  5. Run in shadow mode: the agent drafts, humans answer, you compare.
  6. Go live on a slice of traffic, with disclosure and an easy route to a person; review a sample weekly.
  7. Expand only when metrics hold, and keep a way to switch the agent off.

Frequently asked questions

What is a customer service AI agent?

Software that handles requests in chat, email or phone calls by acting through connected tools, such as looking up an order or issuing a refund within limits, and handing off to a person when it cannot or should not decide.

How is it different from a chatbot?

A chatbot answers from a script or knowledge base. An agent also acts in back-office systems and is judged on whether the task was done correctly.

Is a company responsible for what its AI agent says?

In Moffatt v. Air Canada (2024), a British Columbia tribunal held the airline responsible for wrong information from its website chatbot. Other jurisdictions and facts may differ; this is not legal advice.

Do customers accept AI agents?

In Gartner's early 2026 survey, half said interactions are easier with generative AI and 87% said access to a human is essential; in its December 2023 survey, 64% preferred no AI in customer service. These measure opinion, not outcomes.

What about AI call center agents on the phone?

The same questions apply: what it can do, how it hands off, how you measure it. Pricing is covered in AI receptionist cost.

In short

A customer service AI agent earns the name by completing requests in your systems and handing off cleanly, not by sounding fluent. The sourced evidence shows faster handling, mixed customer ratings and, in one tribunal decision, an airline held responsible for its chatbot's wrong information. Define "resolved" first, ground the agent in one approved source, and expand on measured results. To scope a pilot, talk to us.

Sources

Scope your pilot