Author: Steffen Meyer, Mobile Marketing Content Specialist
Imagine, you launched a campaign and sales are going through the roof but you can’t prove to anyone that this was actually your work.
Not only will this probably bruise your ego, it’s a nightmare for your company as well. Reliably measuring the success of campaigns is fundamental to fine-tune your overall marketing strategy and thus increasing revenue and profit.
That is why you need control groups.
Control groups are a group of people that are shielded from marketing measures: They don’t receive your well-crafted campaigns or see your creative ads. This way, you can compare their data with the data of users that are exposed to your marketing and thus measure (and prove) the effects of your actions.
Categories and size of control groups
There are three categories of control groups:
- The global control group is totally shielded from all your marketing efforts whatsoever. It won’t receive any campaigns you send out to your users.
The global control group should always have lower conversion rates than the rest of the users who are exposed to your marketing. This shows that your ads, mailings etc. are working in general. If the global control group achieves the same conversion rate as all the users that are exposed to your mailings, ads and so forth, your marketing efforts are not contributing anything to the success of your product. If the global control group is even performing better than the rest of the users, your marketing is detrimental to success and you should change course immediately. - The campaign control group is shielded from a specific marketing campaign. It won’t receive any content of this specified campaign but still those of others.
An effective campaign should have significantly higher conversion rates than the control group who doesn’t receive the campaign at all. If the control group performs head to head or even outperforms your specified campaign, it’s not working or even deterring. You should stop and/or alter it before spending any more money on it.
- The variable control group is shielded from a new variation of an advertisement, feature, message, or other user experience. It won’t see this new variation but the already implemented version.
If the variable control group performs worse than the new asset, implement the new one, if it performs better, keep the old one. If it’s a tie, you may choose which asset should stay up since it seems to be indifferent for users.
Each control group consists of a certain randomized percentage of users. The larger the sum of users, the lower this percantage has to be. For 10,000 customers, a control group of 5% is sufficient, while for campaigns targeting less than 2,000 customers, it’s a good idea to use 10% to 20% instead.
Additionally, you need to adapt these proportions depending on expected response rates. When users are likely to react, you can lower the percentage, when users are unlikely to react, you should increase the percentage to obtain statistically significant results.
Control groups aren’t always necessary
Keep in mind that you don’t need a control group for every campaign. Here are some examples:
- Non-trackable campaigns don’t need a control group because there won’t be any data you can compare.
- Reminder campaigns that re-target customers who were part of an initial test group don’t need a new control group. The old one is still valid.
- Transactional campaigns like a purchase confirmation, an invoice etc. should be sent to all customers. You may implement variable control groups when you want to test certain variations, for example in messaging.
- Educational campaigns are often sent without a control group because marketers want to inform all their customers about new features or products. While this is understandable, keep in mind that educational campaigns may also influence customer behavior and you could only prove so with setting up a control group.
Be sure to subscribe to our newsletter, so you don’t miss it. Also join our community on LinkedIn where we always share new material and blog posts.
💡 Knowledge sharing is at the core of what we do. Get our Marketing Master Map and learn how to make apps succeed in the competitive mobile landscape.
Helpful Links:
