How Vodafone’s Custom Scoring System Clarified Outcomes

Vodafone is a leading telco that operates in 15 countries, with over 300 million customers.

As Vodafone puts it: "We believe connectivity is a force for good. It’s an essential part of our lives. If we use technology for the things that really matter, it can improve our lives and the world around us."

The following interview takes place with the Digital Experience team from Vodafone's Consumer Product and Services division.

Dscout’s Customer and Community Marketing Manager, Colleen Pate, sat down with the team to learn more about how they blend design, UX, and research—and how custom metrics changed the game.

Ashton Snook is Head of Design and Research.

Georgie Thompson is the Lead UX Researcher.

Nick Lockey is the Lead UX Strategist.

Jump to:

• How research teams work at Vodafone
• Building the right thing vs. building the thing right
• Prioritizing discovery work, concept work, additional feature development, and ensuring customer value
• Creating Vodafone-custom metrics (why and how)
• How stakeholders reacted to the custom metrics and scores reported
• Wrapping it up

Colleen: How do the research teams work at Vodafone?

Ashton: Within our area of focus (Consumer Products & Services), we essentially have two research divisions. The first is UX Strategy, led by Nick Lockey. This team predominantly focuses on answering the questions at the beginning of the pipeline. In broad terms, their focus is to answer, "Are we building the right thing?"

The second team is UX Research, led by Georgie Thompson, their focus is largely to answer "Are we building the thing right?" Both are opposite sides of the same coin, enabling us to look at the “why” and the “what” behind the idea, design, or strategy.

Tell me more about those dueling questions: are we building or designing the right thing and then, are we building or designing the thing right?

Nick: We had a lot of discussion about how to visualize this. It's two things at one level. It's a linear way of imagining where the products sit and where all the research studies sit.

But then as you can see on this chart below, that line loops back on itself and twists and turns, so it's not an A to B thing. This is because the products and services we work on often move back and forth between strategy, where we're trying to make sure we're delivering something valuable to our customers, and usability, where we're trying to get the execution right. Each study provides new levels of insight which can often put a new spin on things or might require tinkering from either or both of our teams.

I think the best way to understand the scale of our work is that we're a product design team, so a lot of stuff happens before we do our work. A lot of business casing, a lot of exploring audiences and needs, but when it drops into our realm, it's like, how do we turn that into a tangible product that actually gets the market?

If you think of a left-to-right scale, the very left end is more my domain, which is, are we designing the right thing? What could this product be? What are all the different variations this thing could end up looking like? Where could we take it? What could we do? What are the different value propositions?

As we refine and iterate that, it kind of moves along the scale until we build that confidence over what is that itch we're trying to scratch for the audience, and how do we shape it?

Then as it gets to the right, which is, how does that product come together to get to market? That's where it goes to Georgie. So we know what this thing is for now. How do we optimize that to do the best it can to make the biggest impact for our customers?

We're a very customer-centric department, but what's interesting is that at the start of the journey on my side, we have to be very sympathetic to how that aligns to the business needs. How does that align with the commercial model? How does that align with the engineering teams and what they can build? We can't deliver value for our users if we can't get that to market.

That means that we work with our stakeholders and other people around the business to make sure that our vision for making something fantastic for the customers gets past all the stuff it has to do to make it viable as well.

Georgie: My realm is around evaluative research. We've had the great work that Nick and his team have done to scope out the questions that we might have and check that we're creating something that our customers want or need, and it has some value to it because again, you can have beautifully designed products, but if they're not adding any value for people, then it's irrelevant.

On the flip side, you can have a great idea, but if it's not usable or it's not understandable to our customers, then it's not going to work. You have to hit that sweet spot and that's where I come in. Sometimes pre-launch, sometimes post-launch, and sort of everything in between.

It's checking that, okay, we've got this great idea. Now what? How do we present this in our designs? How do we ensure our customers can fulfill whatever experience or objective we need them to do?

I think what's key here and why we've got this sort of loop design is that it's like Nick was saying, it's not linear.

Just because it comes to me towards the end, I might discover something in my research that brings out some new questions that we'd not thought of. New ideas come up from our users, and I think that's something we need to explore further. So that might be something I pick up or it might be something that I loop back into the strategy team and they then explore that further.

It’s the same with Nick. He's looking at the discovery side, but there might be instances where he sees that we can fast-track this. Let's just test this now because we know it's a good idea for whatever reason or actually, we don't need to spend lots of time workshopping this, we just want to get some answers in terms of how people use this or work with this.

It loops around. It doesn't just stop or start with Nick and end with me. It crisscrosses and, as Ash will probably explain in a minute, product design sitting in the center there is integral as well.

Ashton: The graphic here massively simplifies the complexity of the journey. It might be better shown as spaghetti in reality, and yet no matter the actors involved, it communicates the core truth. Our people focus on these two core questions and work together to translate the insights they find into effective design solutions.

Is there a way your team focuses and prioritizes discovery work, concept work, additional feature development, and ensuring customer value?

Nick: Those stages of research are probably quite familiar to a lot of people here. Going back to that sort of non-linear thing, we're not a huge team and we're jumping between projects and different products and different stages of products all the time.

So it might be the project that we're doing today and the one we're doing next doesn't necessarily follow on from that one. We might be jumping on the next biggest priority for the products and services team. We're always jumping between the two, but at the same time, we're quite organized as well in terms of understanding the roadmaps and the questions that we pick off at any time.

We're often jumping between very early discovery to things that are near-launch or post-launch or something in between. We're sharing insights across the team continually. We have lots of meetings and ceremonies between the team every week or every month where we come together and we showcase what we do. We share what we do with the wider business as well.

It’s interesting because, partly by design and partly by necessity, we've never really followed the model where we'll have a researcher embedded in a product. While we were establishing the team, we had to be a bit more fleet-footed and help lots of different teams pick off the most important questions at the time.

Over time, that became a bit of a secret superpower for us because we're jumping between different products and different stages. A lot of the time, these teams are heads down in terms of just trying to answer their particular product questions and moving the dial on what they're doing and doing a really good job on that.

We've ended up being in a position where we can often take a bit more of a helicopter view on it to see what various teams are trying to do and how they're all solving problems. We can very often go, “There's a team over here that has come up with a solution that will work brilliantly for this product, or two teams are trying to answer a similar question and we can bundle that together into one study and get more value for them.”

Not having that more traditional model where a product team will have an embedded researcher lets us move around a lot more and overcome a lot of the bottlenecks as well. Sometimes, you'll have a lot of work to do on one team and then things will slow down while they're putting that into engineering or something.

It just means that we can move our research resources around and make sure they're always really delivering value for whichever team needs it the most. Also, and this is something Georgie might explain a bit as well, it helps overcome some of the attachment we have to the products and we can be a bit more objective.

Georgie: Being embedded within a team has lots of advantages. You get to know your product space, and the stakeholders that you work with. But that can also cause you to become potentially less aware of any problems that your team might be facing or any challenges.

You might start taking things for granted or not see the bigger picture. Having the ability to work across teams, you can see how different product squads are organizing themselves and how they're managing things.

It allows you to take the best bits from all and cross-pollinate that across all the teams, rather than getting stuck in your ways. That's obviously not always the case if you're embedded, but it can be that. We find that it's nice being able to work across squads.

On a personal level, it gives us that experience of working with lots of different product teams and different questions that we're trying to answer, because you can get stuck answering similar questions, particularly on my end, when we're doing evaluative work, answering similar questions over and over again.

We get to work with different product challenges and with different personalities. I think it keeps it interesting. As well as the great reasons that Nick mentioned from a business perspective, I think personally, we find it an enjoyable way to work as well.

Why was creating Vodafone-custom metrics necessary for the business and how'd you start?

Georgie: We reached the stage as our research department matured, that there became an obvious desire for clear and easy-to-understand insights, and that then became more of a requirement.

We found ourselves with this challenge of selecting and developing UX metrics that allowed us to assess our products more easily. We wanted to create a common language with our stakeholders across the product cycle. This goes from discovery to validation.

We wanted that consistency to be present across the whole cycle. Not just, I'm asking certain questions at one stage and Nick is asking something completely different at the start. This allowed us to benchmark and have some consistency when we ask our customers for their scores at the end of the session.

We often find that when we get to this wrap-up part at the end of our sessions, this is typically where we get interesting insights. This is where customers can come alive. They almost feel like they're not in a testing scenario anymore, which we try to help them feel throughout, but it's not always possible when you're being recorded on the screen and you're having questions asked at you and you're using a prototype.

When that stops we get them to think, okay how did you feel? How was your overall experience using whatever it is or with this idea? That's when they get talking. We wanted a way of capturing that because while we always did a wrap-up previously, we didn't necessarily have a way to measure that experience.

That became more of a requirement to measure consistently. But don't get me wrong, we’ve had a few doubts as a team at first about using metrics, particularly for qualitative studies. We typically talk to small cohorts of users in depth, but it's smaller sample sizes.

There was some concern if doing a score for this type of work is the right thing to do, but we've been doing this for a while, at least a year. I think the metrics that we've established that we're going to talk through shortly are working well for us.

We're now advocates of it which, if you asked us 18 months ago, that might not have been the case. So I think you have to give these things a try, and hopefully what we've learned can help others potentially use them for core studies as well.

Ashton: As they said, we saw it as a bit of an opportunity and challenge. I think Nick's probably the best person to talk through this. But, as I said before, we were surprised how positive an impact these metrics have had on our workflow and on how effectively we've been able to engage with our stakeholders.

While it's great to have the level of depth we provide in our research reports, and through the video content on Dscout, these metrics provide our stakeholders with something simple and easy to digest as a bite-sized takeaway from the research.

They've also helped us, and our designers, to engage with our senior leadership team too, especially around investment opportunities where we can now convey the impact of our designs in simple terms, which is really exciting.

Nick: Yeah, I think what's probably my favorite thing about these scores is that they actually came out of a design-thinking process. They weren't just born like this. We started off with the customer effort score, CES score, which is one that's fairly well known.

We thought out of all the metrics we could use, CES was the right one for two reasons. It was being used around parts of the business already, so there was already some buy-in for it. It gave us a score that enabled us to then go on and create very designable problems and solutions from that. The score felt like a natural fit.

We started off using the customer effort score and we trialed it on a few studies. It was great. As Georgie says, we use it at the end of talking to a user for 30, 45, and 60-minute interviews. We have a section at the end where we ask the question, “Based on the design you've seen today, how easy would it make it for you to achieve whatever goal we're testing for?”

We were seeing it was working very well as a point in the interview to help them articulate how well the design was working. It was really nice. They'd reflect on it and give us something that we could very easily go back into the research notes and find their summary. We noticed the limitation of using the score by itself. The score didn't separate whether the value proposition in the idea was good, and whether the design was good.

We were finding situations where we were fishing in the right water and we would hit a really good pain point for them, but they might not like the design. Or they liked the design, but the pain point was wrong and we couldn't separate it with just the CES Score.

We then created a custom score. There might be other scores out there like it, but we used the same seven-point scale and we called that a Customer Proposition Score. So the idea at that point became that we asked for the CES score and the CPS score proposition score separately. That's two questions at the end of the interview.

We've asked the proposition score first, so that question was irrespective of how it was designed and executed. Would a product or service like this solve a genuine pain point for you?

The idea for that was, we can capture that sentiment of whether we’re fishing in the right water and whether we're onto something with the value proposition. And then we'd ask the CES score separately. Would the way that we've executed this design make it easy for you to complete your goal?

We could then see whether the design in itself had strength. There's lots of situations where we were answering the pain point, but the design needed a tweak and we could definitely look at that then. There were other situations where we didn't have the right value proposition, but there was something in the design that people really liked.

Going back to that cross-squad thing again, there's bits of designs that we can recycle for this product for future studies with a different value proposition, or it might work for one of the other teams. Or it shows that the design worked, so it did double the work.

We got to test out different design patterns and at the same time, we could use it as part of our wider design system. It was a very efficient way of doing it, but then we noticed a third thing. This is where the continual iteration of these scores came out that, like Georgie said, we're predominantly a qualitative team.

We do a lot of talking to small cohorts in a lot of depth, and we were finding that was part of the interview. We were often building up quite a good rapport with people.

We'd spent a bit of time with them, and then at the end when we asked for these scores, particularly the effort score. They would give us a great score because we'd explained what these products were, and they'd got to like us a bit during the interview. But really we were looking back going, ‘They stumbled over that bit and they misunderstood a core part of the value proposition.’

So we introduced this third score, the usability expert (UXS) score. That is a heuristic measure that we will take after we've done each interview or after a bunch of interviews. The researcher and usually the designers who have been taking the notes as well, then look at the scores that the customers have given us.

Through our UX expert eyes, we see whether we need to adjust any of those scores. So for instance, the customer might have loved this product, but we actually know that they fundamentally misunderstood something in the copy or the design.

It might be they've given us a low CES, whereas actually what they disagreed with was the proposition, and they just got confused between the scores. So it gives us a little wiggle room there using that heuristic expert view to adjust the scores as well.

We report all three of these back to our stakeholders as well, so they've got a really clear sense of the value proposition. Did the customers say they liked the design? Are there any sort of little tripwires or gotchas that we've picked up as a UX team, where we have to clarify some of those scores or maybe give a little bit more context? These three together have worked really well.

How did stakeholders react or interact with the metrics and scores you reported?

Georgie: I can talk to this in the sense of how we're educating our stakeholders in using these scores.

Now that we've been doing it a while, everyone's certainly a lot more familiar. But I think we found at the start—or I certainly found at the start—that there could be an over-reliability on the scores themselves, rather than the sentiment and the patterns that we were seeing. I often will run iterative work where we test things perhaps two, three times, change the designs, and test again.

We would use these metrics at the end of sessions and we would sometimes find that scores, let's say 5.8 in the first round, and 5.6 in the second. We know as researchers in our team that those are both good scores. We shouldn't have huge concerns.

They're both scoring well, but when you present that side-by-side to your product manager or anyone else, it can feel a bit alarming. They think, ‘Oh, but we've been working hard on these designs and it's gone down a little bit.’ I think where the education happens here is, it's not so much the exact score, again. Like we said, we typically speak to a small cohort of users.

On another day, you could have eight different users and the scores given might have been 6/7 or 5.7/7—they are liable to some change, particularly when you're taking averages. It’s been an education that it’s just a way of showing that something's either working well or needs a bit of improvement and not necessarily getting bogged down on the exact score.

I needed to learn that myself as well, because I would also have the same question of, oh no, I from professional experience, think this has got better and I've actually witnessed people using this really nicely, but our scores don't necessarily reflect that exactly, or they're the same.

People get a bit disheartened that they haven't gone up, so I think that's been a learning for us on how to communicate the scores. It's not the scores, it's how we use them and how we communicate that back to our teams—to make sure they are valuable to our teams, and not just a number on a page. We want to make these usable and allow our stakeholders to know what to do with them.

For me personally, that's been a learning that I've found and I've been tweaking myself over the last year.

Ashton: The really cool thing is research serves a role to inspire and drive conversation. The scores have been very digestible and they're very clear, and they allow us to facilitate something that people can take away without having to think about all the nuanced detail.

Nothing's empirical—no research metric in the world can say for a fact—that this is going to be absolutely bulletproof. What it does is, obviously, with great confidence allows us to make more informed choices. But because it's bite-sized, if you dive into details, see the comments, see task completion, etc., that engagement has been very well received.

The simplicity has been key.

Nick: The best thing for me is it's given us a shared language with the stakeholders and the users as well. What we're seeing increasingly with the stakeholders is that it's far easier to articulate the difference between a problem with the value proposition and a problem with the design.

When we bring these scores up on screen when we're talking on Dscout to our users, it gives us that moment in the interviews and often the video clips that we show where we can hone right in on where the users are discussing these scores. It helps us bring that to life for the stakeholders as well.

We're all a bit worried when, as Georgie said, we were a bit reluctant at the start because we were like, ‘Are these scores going to be used as a stick to beat us with if they go down? Or are they going to be misinterpreted and scaled up?’ We haven't really seen that.

Everybody's been smart and switched on about exactly what they're there for. It’s not really there for the score in itself, though, it is useful to track it. It's more that by asking the question through these metrics, it condenses the focus of a conversation or making a point about the product that we're doing.

It’s been more useful as that shared language in a way than it has as a score.

Georgie: We can't be delivering pages and pages of information every time we do a study. It's just getting that point across quickly in a digestible, clear way for them.

It helps us condense that wrap-up and focus our attention on what's important here. We've got lots of great work that we've been doing, looking at designs or prototypes—but now what's our focus and what are our core questions?

What were our objectives for this? Let's summarize those. I think these metrics help us get to the core of that in an efficient manner for us and our team.

Wrapping it up

Uniting multiple teams across departments takes a tremendous amount of effort and experimentation. Creating a shared language with stakeholders through customized scoring enabled Vodafone to improve on the product in ways they didn’t think were possible.

Interested in checking out the entire interview? You can watch it here.

Build deeper connections with users and spark stakeholder engagement

See how Dscout can connect you with the right participants, gather rich in-context data, and help you drive research impact throughout your entire organization.

Schedule a demo to learn more about our platform and team.

You may also like …

• How KEEN Built a Continuous Discovery Process That Aligned Product, Design Marketing, Merch, and Insights

• How SiriusXM Drove Product Innovation Through Bottom-Up Research

• How to Engage Stakeholders in Research for Better Outcomes