Using a Prototype to Test an Assumption in a Messaging Application: Product Management Case Study

Executive Summary/TL;DR

To validate investing time in building a collision detection feature, we used a very simple prototype — showing collision — to confirm validity of our approach and justify investing more resources into it.

Introduction

Disclaimer: For the sake of confidentiality and anonymity, all details contained in this case study (especially PII such as names, addresses etc.) are fictitious and don’t contain any details about the respective client. If you’d like to learn more, reach out to me directly.

This case considers my work on a messaging-based application that facilitates conversations between larger brands/businesses and consumers across multiple channels.

These business would receive messages (cases) from customers, and have one to many customer service-type agents responding to and dealing with those cases from a central case queue.

Consider an interface that looks somewhat like this — **left** is the case queue, **right** is the case detail view. Image credit: my own rather basic Miro design skills.

Problem Statement

At one point, I was approached by one account manager in charge of multiple large accounts; they were eager to pass on consolidated feedback:

“A few accounts have complained that they cannot track each other working on cases simultaneously, so it has happened a few times that two agents will work on the same case, which leads to unnecessary duplicate work, and in the worst case, to two answers (sometimes with conflicting statements) being sent, which makes their customer-facing presence look really bad.”

Here’s an example of the problem. Image credit: my basic Miro skills again.

Reproducing the Issue

First, I wanted to make sure I understand the problem correctly:

I logged in as two users of the same account
I located and opened a case on both accounts
Based on the refresh infrastructure we were using, it was well possible to even send two replies and not notice it until a couple of minutes later.

Understanding the Dimension

I went back to check on how substantial this issue was — my AM contact provided me with details on which accounts raised the issue, including a few examples.

We defined the problem clearly:

“Accounts with multiple agents working simultaneously cannot see each other in the queue, leading them to do duplicate work (waste of resources) and send duplicate answers (bad customer experience).”

Then, reached out to other AMs to garner more input and support for this problem — a few others did respond and flagged more accounts having mentioned this issue.

What about assignment?

We did already have a case assignment feature (i.e. agents assign themselves or someone else a case from the queue, to indicate that they’re working on it) though — so my next question was why they’re not using it.

Back to the initial view — the assignments are highlighted for you here. In this example, consider that the open case is assigned to Jonas, who might be on leave, and might have not handed it over correctly. Image credit: Miro, myself & I.

“Some accounts do use assignments, but as their teams work in shifts, cases often have to be re-assigned, and updating assignments (or rather reflecting updates) can take some time.”

→ My keyword here was “some time” — the user need was for updates in (near-)real-time.

At this point, you might think — “Well, let’s just update assignments more rapidly across all users?” — there’s more:

“Also, other accounts completely disregard assignments as they don’t see value in it. They just leave all cases unassigned and work through them chronologically, first-in-first-out (FIFO) style.”

This image is a bit of a trope, I know. But this is a rather spot-on example, isn’t it? Image credit: Reddit.com

→ This is the classic “Design vs. UX” joke (see above) in the flesh:

If you allow your users to use your product in an unintended way, don’t be surprised when they do so.

We can’t force those accounts to use our assignment feature.

Moreover, as established before, the assignment feature itself doesn’t (fully) combat the problem at hand.

Solution

Solution Design

Here’s the problem again for review:

“Accounts with multiple agents working simultaneously cannot see each other in the queue, leading them to do duplicate work (waste of resources) and send duplicate answers (bad customer experience).”

Let’s break this down:

Agents may either be in a shared queue view with other agents
Or have a specific case open that they’re working on

Accordingly:

Agents in queue seeing agents already in a specific case would indicate “this case is already being addressed by someone else, i.e. no need to pick it up”
Agents in a specific case seeing someone else editing the same case can connect to their fellow agents and solve a possible dispute or misunderstanding.

So, the proposed solution was:

Indicate a “[NAME] is editing” in the queue view” to flag in-progress cases to agents
Indicate a “[NAME] is editing” in the case view to flag possible disputes to agents.

Here’s an example of the full solution for points 1 and 2 mentioned above. Image credit: still myself.

Solution Specification

Implementing this solution would be much less straightforward than it might’ve seemed, however:

We didn’t have any infrastructure or service that tracked user activity on the front-end layer (much less in real-time).
We didn’t have any existing front-end components for this feature.
We also hadn’t done any research on UX details — how long should this be displayed, at which position etc. (except for best practices).

So it would be quite a bit of work to do this properly.

But as we couldn’t know whether this would actually solve our users’ problems, we weren’t willing to go all-in on this.

The Prototype

As I’m a fan of the RAT (Riskiest Assumption Test) concept, I considered our ‘riskiest assumption’:

IF users see others busy on a specific case

THEN they can prevent or mitigate duplicate work.

Luckily, my lead front-end developer had been toying around with WebSocket, and was eager to set up a prototype — so our first target was:

Can we transmit information from one user’s browser to another user’s browser in (quasi-) real-time?

He built this within a few hours, on a test instance, only in the case detail view using basic text to be displayed as a control.

It looked rather bad design-wise, but IT WORKED! So step 1 was done.

It looked something like this. Image credit: still myself.

Next, we wanted to (carefully) ship this prototype to the production environment, and validate whether this would actually help users.

So, we set up a minimal version — basic UI/UX, minimal functionality:

When keystrokes in a case view are detected, a message is sent out to all other open instances of that case view to display a small message at the bottom right: “[NAME] is editing”.
The message should live until 1 minute after the last keystroke, then disappear.

I was actually willing to settle for dropping the [NAME] requirement and just display “Someone is editing” if that would’ve been too much additional work, but my lead developer had figured out an easy way to solve this!

This of course took a few days of development and was implemented over the course of a sprint, but nonetheless didn’t break the resource bank.

This is what the prototype — and for the most part, the final implementation — looked like. In this example, Ignacio would have seen that Larry was already working on this case, and could either just pick the next case, or reach out to Larry via some other means of communication (an IM like Slack, over the phone, or in person if in the office) to settle who should work on it. Image credit: still myself.

Results

We then shipped our collision detection feature, and decided to first wait for users to notice before we told them.

Two accounts of my main AM contact actually reached out proactively and commended the update, saying they were happy they now could resolve these disputes proactively!

We then distributed release messaging to all other accounts, and received positive feedback overall as well.

We also went on to replace the WebSocket prototype with a ‘proper’ implementation (as the prototype ended up not scaling well), fully integrating this feature with the platform.

Conclusion

Using a prototype, we were able to validate our assumption, iterate rapidly, and solve two nascent customer problems — resources wasted on duplicate work, and bad customer service reputation (due to duplicate and/or conflicting replies).

With this very iterative and minimum-viable approach, we were able to verify product-market-fit (PMF) early and invest additional resources with confidence. Luckily, this approach has become more common these days, but wasn’t pervasive at the time.

Recommendations

Make sure you understand the problem properly. This is crucial — there are often a lot of solutions to a problem, but you don’t have the time/money/nerve to try them all. Actually understanding the problem will guide you well.
Next, move fast — ideally prototype. Full solutions often get overcomplicated by the need to integrate with all kinds of feature and technical debt. Adding a very small prototype implementation is a low-risk/effort, high-impact way to validate your plans.
Make sure the prototype STAYS a prototype! The prototype should be part of your plan to justify building a proper solution. Often, people have been tempted (successfully!) to just keep using the prototype (“well, it works just fine, and we could do other stuff”), but that will destabilize your application and cause technical debt in the future.

Thank you very much for reading!

If you’d like me to help you with a similar challenge you’re facing, I’m a Freelance Product Manager & Consultant for hire — feel free to reach out to me at hi[@]bertrandrothen.com.