Access required
This case study is password protected. To view it, please enter the password below.
To request access, please contact claremnguyen@gmail.com
Migrating the Content Library to Google Vertex
The Content library was the "brain" behind Drift's AI chatbots. Customers uploaded their websites and files so their bots could answer visitor questions accurately and autonomously.
In late 2024, we rebuilt it on Google Vertex to fix accuracy issues with ChatGPT.
Drift / Salesloft
Clare Nguyen, Bionic Team
Product designer
Competitive analysis, usability testing, prototyping, UI design, UX design, UX writing, wireframing, interviewing, surveying
December 2024 - April 2025
Background
When Drift introduced its first iteration of generative AI chatbots (also referred to as the "AI response node" or "Bionic chatbots") in 2023, we launched the Content library as a way for users to "teach" their chatbots about their company and their products.
It was the "brain" behind our customers' bots. Any websites, URLs, or DOCx/PDF files they uploaded became things our AI could learn from, so bots could answer visitor questions on their own and free up live reps for support tickets and qualified leads.
Results
The rebuilt Content library kept the familiar table view customers already knew, while the move to a full-page layout gave us room to scale as we bring features back.
The migration to Google Vertex doubled the bot's acceptance rate among early access customers, from 40% to 80%, and cut intent classifier errors by 60%.
By the end of February 2025, the acceptance rate was still trending above 80%, a sign that the improved accuracy held up after launch.
80%
In overall testing acceptance
60%
Decrease in intent classifier errors
3%
Decrease in need to test
Problem
Over time, the Content library became quite robust, supporting everything from entire websites to exclusions, languages, and meta tags. But there was one big, glaring problem: ChatGPT was not very good at parsing and reciting data. Our customers noticed that their chatbot would often get details wrong, even when it pulled from the correct sources.
Pivoting with a proof of concept
After trying to correct the LLM as much as possible, we set our sights on Google Vertex in December 2024, and early engineering tests showed a huge spike in performance. Unfortunately, a simple plug-and-play swap wasn't possible. We had to start over.
On a tight time crunch, we scoped the MVP down to the bare minimum our customers relied on: entire websites and files. We knew not everyone would be happy about losing features, but we believed the improved performance would change their minds.
Design
I opted to keep the Content library as familiar as possible, with the same table view and the ability to click into content for details. The biggest change was replacing the drawer with a full-page route. We'd already struggled to fit all the content details into a drawer, so even while cutting features, I designed with scalability in mind.
The biggest hurdle was making the migration as painless as possible. Because Vertex limited how many URLs could exist within a library, I came up with a migration wizard that let customers pick through a few options. We also rolled out in stages, with early access first and then general availability, so I designed multiple stages with changing copy and worked closely with engineering to keep them in sync.
Midway through, we discovered that customers had to verify their domain and approve our association request before any website could be ingested. This was the slowest part of the migration because it relied on customers to act. To minimize delays, I added clear messaging in the Content library and email reminders on a 3, 5, and 7-day cadence.
Results
In the end, our hard work paid off! Just from our early access customers we observed an increase in the bot’s acceptance rate rise from 40% to 80%. We also noticed that there was a 60% decrease in intent classifier errors.
By the end of February 2025, the acceptance rate continued to trend more than 80%. We saw that there was a decrease in testing (down 3%), but we believe customers were beginning to trust Drift’s AI and saw no need to test as thoroughly as they had previously.
Conclusion
The whole project was quite chaotic, but I don't know if there was anything I would have done differently. The entire team was flying by the seat of our pants and for a majority of the project, I had to design on the fly. I was learning about Vertex at the same time the engineers were.
I will say that the team worked exceptionally well together and I'm proud of our collaborative efforts. Everyone (not just the trio!) worked hard to make this happen and every time we hit a snag, we all rolled with the punches.
Overall, I'm quite pleased with how this turned out. Our customers have been incredibly happy with the migration and we've now set our focus on bringing back the features we had before.