The ‘New’ Coke: Market Research Failure. And …. how to avoid it!
How 190.000 user tests couldn’t avoid getting it wrong.
Helped by a well-aimed marketing campaign and stars like Michael Jackson, Pepsi had been steadily gaining market share in the ’80s. In a market segment that was very threatening for Coca Cola: teenagers and young adults; the consumers of the future.
Coca Cola had to do something and …. They did. They designed a new formula that would help counter the rise of Pepsi. Coca Cola tested the ‘New Coke’ on 190.000 (!!!) respondents before introducing it in April 1985. They said: “We have never been more sure of anything in the history of the company”.
Yet the ‘New Coke’ failed miserably in its mission. How is it possible to fail in such a spectacular way after testing so much and more importantly: how can you do your research to avoid failures like these?
What if it was your own money?
I often hear: ”yes we checked that, or ‘yes we researched that’ ‘There is a positive business case’ let’s do it already.” Shouldn’t employees and managers be quite a bit more critical and curious I ask myself? As if they were spending their own money? Is ‘a business case’ enough? In many companies, you can become a ‘hero’ by getting things done. But are they the right things? Is a single user test with 6 people enough? Would you cross a very busy street without looking twice? I don’t think so.
If you look a bit closer at research you soon realize that there are several ways to do research. So it should be clear to everyone that there is a difference in ‘confidence’. For example:
- Showing a prototype to a random person you know or meet
- A screenshot in Usertesting.com for 12 respondents that claim to fit a certain profile
- A professionally guided research with 12 handpicked clients or people that have recently bought a lease car that click through a real demo or a test website.
- Real behaviour: A set of a/b test that shows a ‘treatment’ performs significantly better or a test in a test shop/country/demographic
Tested 190.000 blind tests
Coca Cola was so sure of the New Coke because they tested it on 190.000 Coca Cola and Pepsi Cola drinkers. It beat ‘Old coke, Pepsi coke in the tests they had run. So when they introduced the New Coke it is was with a lot of confidence. But, was the test a good approximation of the way people consumed Coke. It turned out. it wasn’t.
New Coke was launched, people were curious, bought a few bottles, took it home. And …. next time they went to the shop they bought ‘Old Coke’. Coca Cola had to roll back the whole introduction at a cost of hundreds of millions of dollars. They received up to 5000 calls a day with people expressing their anger and sadness about the ‘New Coke’.
The post mortem
The post mortem* showed: they had run the ‘wrong’ test. They had given participants a small cup of ‘Old Coke’, ‘New Coke’, and Pepsi. A so-called-’Sip-test’ The new coke was much sweeter which put it ahead in the test. But drink a liter of it or 2 liters as many Americans do and it was way too sweet. This begs the question: why did they not test it in a certain area, number of shops (real transaction), or even test consumers but over a longer period? And with real ‘1 or 2 liter bottles’ of Coke? Lessons learned: Always wonder how representative your test environment is of the real world scenario* and do multiple types of tests.
There is a happy ending though: the Americans discovered how much they were attached to the ‘Old Coke’ and soon after the fiasco the ‘Old Coke’ was US’s bestselling soft drink again!
Testing Confidence Framework
How do you know if your tests are to be trusted?
The short answer to the question “ How do you know if your tests are to be trusted.” Is “you don’t’” There is no 100% secure answer to that question. That is why start-ups fail, pivot, and why large companies find it so hard and costly to innovate.
Take the Starbucks test which is much used by start-ups; Show a sketch of your product or a prototype to the person next to you in Starbucks. So what if he/she says: “Yes I would definitely buy that.” Does that mean your set for success or is that person just being nice? After all: there is a difference between what people actually do and what they will do. I am not saying the feedback is not valuable. It is! However, I wouldn’t bet my product or company on it. Saying: “yes we tested that!”.
Without going into all user tests that are available I would like to introduce a simple framework that can help you to ‘rank’ how secure a research method is. I am using it to asses if new features, services or products have been properly tested. By awarding points in these 4 categories we can assess the confidence in a test or sequence of tests:
- Real or fake transaction?
- Prototype/Lab or Real situation?
- Real ‘blind’ significant test
- ‘Quality of the subject selection
Add up the scores in each category and yes, now we can say with confidence ‘We tested that!’
- High confidence 65-100 points
- Medium confidence 30-65 points
- Low confidence <30 points
- Just opinion /Common sense
Real or fake transaction?
25. ‘Real’ transaction/lead
10. Click through in Live product (to fake page)
5. End result in Clickable demo (lab setting)
1. Interview question
Prototype/Lab or Real situation?
25. Live environment
10. Clickable prototype
5. Static screens/designs
1. ‘Would you be interested in?’ (Opinion)
Real ‘blind’ significant test
35. Proven higher $ or margin/ in pilot market
25. A Repeated live A/B test 99% confidence
15. A single live a/b tests 95% confidence
10. Before/After analysis or Online survey
About the subjects/panel
15. Real customers/real target group
5. Handpicked to be like target group by agency
1. Online self-selected panel (like usertesting.com)
*There are many accounts of why ‘The New Coke’ went wrong. Some claim it is a branding mistake, others a PR mistake. The source I trust most is Malcolm Gladwell’s ‘Blink’ book (p158) as he is known for the quality of his research.