Coinbase Speeds Up Customer Support Testing with Autopilot System
Coinbase has significantly accelerated its customer support testing process using its Autopilot system. The company reduced a 90-case test from one to two weeks down to just 30-45 minutes. This improvement allows faster verification of the playbooks used by support bots, ensuring they follow expected procedures. A Sept. 21 engineering disclosure revealed details of simulated conversations with these bots, where Autopilot records transcripts and tool outcomes for grading.
The new system involves AI agents helping draft test plans and variations, while a repeatable runner executes the tests. However, Coinbase did not provide any before-and-after figures for customer response times, resolution rates, satisfaction, or errors related to the process. The company also clarified that support bots can check account states, perform limited actions, and hand off cases requiring greater judgment to humans.
Human approval is still required for changes to support procedures before they enter production. Additionally, an AI model scores adversarial test conversations, but these scores are not definitive and only inform human review. The full automation loop, from identifying weak support flows to promoting revised procedures, is still under development.
Coinbase also introduced Control Center, an internal platform for managing service access, approvals, audit records, and rate limits across various teams, including support, compliance, legal, risk, and engineering. However, the company did not specify how broadly these controls apply to Autopilot or its support bots.