A strategic management framework for app store experiments
Our Customlytics Experiment Management model for app store experiments is a step-by-step guiding system that balances iterative testing with strategic planning. It is designed for an error-free testing environment and a virtually infinite loop of conversion rate optimization (CRO), by providing:
- A high degree of consistency
- A long-term vision for organic growth
- A sense of priority
- An abundance of test ideas
- A focus on learning & idea validation
- A scientific approach to managing experiments with the goal of idea validation
The model follows 2 phases: Preparation & Execution.
PHASE I: PREPARING FOR AN EXPERIMENT
Step 1 – Research:
Conduct brand & product analysis, and do research into the target users and main competitors. This provides data to generate test ideas.
Step 2 – Hypothesize:
Generate a falsifiable and well-structured hypothesis based on research insights. A valid structure goes as follows:
The test needs to be well-planned and conceptualized before execution. This requires proper elaboration on the hypothesis. Elaborate in the following 3 steps:
- Develop a story
- Establish a storytelling approach
- Create a storyboard
Step 4 – Brief:
Generate a creative brief that instructs the artist well enough for them to materialize the test variants. The artist can be a graphic designer, copywriter, animator, etc., and the brief should be adjusted in accordance with their types of work. Furthermore, ASO-related briefs should entail particular specifications to make sure the test assets meet Apple and Google’s requirements.
PHASE II: EXECUTING THE EXPERIMENT
Step 5 – Produce:
The preparation phase should already determine what to create and how to create it. In production, the actual design or copywriting work may begin.
Step 6 – Test:
Real execution may begin as soon as test variants are produced. Execute the experiment by:
- Picking the right testing method.
- Choosing a suitable testing platform.
- Deploying variants to the platform, following conditions of the chosen method.
- Beginning the test.
Step 7 – Measure:
Compare the Control variant’s conversion rate (CVR) with the test variant’s to measure test results, then validate or invalidate the hypothesis, and finally make sense of the results – with critical reflection, not assumptions.
Step 8 – Handle the results:
There are 3 standard protocols to follow for the treatments of the results at this stage: Scale the variant/test, scrap it, or adjust it.
- Scale a convincing win by expanding the variant to more languages, assets, etc., while keeping the same strategy. Scaling starts from production – making a growth loop for CRO.
- Scrap a loss that shows little to no value and is beyond salvation. Either revisit a previous step in the Preparation phase or restart the whole process and do the research again. This is a feedback loop that at least guarantees to learn in CRO.
- Adjust anything in between, as long as it is possible and worthwhile to improve the variant. Change one detail at a time, develop a new hypothesis, then test again, and repeat. This is where the true optimization loop is made – with iterative testing.
If you would like to learn more about our frameworks or need help implementing it, don’t hesitate to contact us!
See you around!