Improving AI Agent Containment Percentage at Chartway: Optimizing Automation & Member Engagement

Improving AI Agent Containment percentage at Chartway: Optimizing Automation & Member Engagement

Role - Conversational AI Experience Designer

Timeline - Q2, 2024 - Q1, 2025

Project Type - Chatbot Design & Integration Updates

Tools - Figma, Posh, Glia, CoPilot, Microsoft Excel, Microsoft Word

Impacting Areas - Technology, Fintech, AI, Chatbot, Member Experience, Team Members

Project Overview

I led a chatbot optimization initiative for Charley, our digital assistant at Chartway. Originally launched to support members 24/7, Charley was underperforming with a containment rate between 75–80% and a high volume of escalations. Through a strategic rewrite of intents, clearer conversational structure, and improved member-facing content, I helped improve Charley’s containment rate to 81–86% while reducing escalations below 15%.

The Problem

When Charley launched in July 2024, many of the intents were written with the assumption that the AI chatbot could perform actions, such as logging in, transferring funds, or making account changes. In reality, Charley was built to provide instructional support only. This mismatch created friction, confusion, and user drop-offs.

Additionally, in March 2025, we expanded Charley into our IVR system. However, we found that our chatbot was still directing members to “call us” for tasks like disputes, which often led to members reencountering Charley on voice, creating a repetitive and frustrating loop, especially for our older or first-time users.

Key issues included:

  • An action-oriented tone that misled members

  • Suggested replies like “Log In” that implied direct action

  • Repetitive or vague fallback responses

  • Escalation rates exceeding expectations

  • Cross-channel friction from Charley Chat, suggesting actions that are repeated on Charley Voice

Key Insights

When users interact with bots, they don’t just want speed; they want clarity. By aligning what Charley can do with what it says, we create a smoother, more honest experience.

And when a digital assistant appears across both chat and voice, it’s critical that messaging sets the right expectations and avoids duplicated effort for the member.

Solutions Implemented

Intent Restructuring

  • Shifted from action-based language to FAQ-style, instructional responses

  • Made all replies more natural and transparent

  • Example:

    • Before: “Click here to log in.”

    • After: “Here’s how you can log in to online banking…”

Improved Suggested Replies

  • Rewrote button text to reflect what would happen

  • Replaced vague options like “Log In” with specific ones like “How to log in”

  • Additional Example:

    • Before: “Submit a title”

    • After: “Steps to submit a title.”

Fallback + Escalation Refinement

  • Cleaned up fallback messaging for clarity

  • Added contextually smarter escalation triggers

  • Reduced false hand-offs to agents

Q1 2025 Update: Chat & Voice Alignment

To reduce confusion between Charley Chat and Charley Voice, I revised several chatbot intents that previously instructed members to “Contact Us” for assistance. Instead of vague or action-based instructions, we created a template response that acknowledges available support options in a way that felt transparent, respectful, and complete:

"If you need further assistance filing a dispute, a team member will be happy to assist you using one of the following methods:

  • Type team member in the chat

  • Reach out to us at [800#] and say 'dispute'

  • Speak with our Video Banking team

  • Visit your local Branch

This reduced the loop of confusion for first-time voice users and supported older members by clearly offering alternatives without redundancy.

  • Intent Restructuring

  • Improved Suggested Replies

  • Updated Experience

Results

Containment Rate

Before

  • 75–80% avg

After

  • 81–86% avg

Escalation Rate

Before

  • 16–20%

After

  • <15% avg

Member Clarity/UX

Before

  • Inconsistent

After

  • Significantly improved

Internal Feedback

Before

  • Frustrated teams

After

  • Strong positive feedback

Reflection

This project reinforced that honest UX is powerful UX. By aligning Charley’s personality with its actual functionality, we improved trust, reduced escalations, and created a more unified, supportive experience across both chat and voice for our members.

What I'd do differently

  1. Conduct a competitive analysis to uncover innovative banking chatbot features and gaps in the market

  2. Deepen member research to understand expectations and prioritize adding secure banking functions like balance checks and transfers, if available, through potential partners

  3. Implement iterative prototyping with usability testing to refine new features and ensure a seamless member experience