A strategic management framework for app store experiments

Our Customlytics Experiment Management model for app store experiments is a step-by-step guiding system that balances iterative testing with strategic planning. It is designed for an error-free testing environment and a virtually infinite loop of conversion rate optimization (CRO), by providing:

  • A high degree of consistency
  • A long-term vision for organic growth
  • A sense of priority
  • An abundance of test ideas
  • A focus on learning & idea validation
  • A scientific approach to managing experiments with the goal of idea validation

The model follows 2 phases: Preparation & Execution.


Step 1 – Research:

Conduct brand & product analysis, and do research into the target users and main competitors. This provides data to generate test ideas.

Step 2 – Hypothesize:

Generate a falsifiable and well-structured hypothesis based on research insights. A valid structure goes as follows:

Step 3 – Elaborate:

The test needs to be well-planned and conceptualized before execution. This requires proper elaboration on the hypothesis. Elaborate in the following 3 steps:

  • Develop a story
  • Establish a storytelling approach
  • Create a storyboard
Step 4 – Brief:

Generate a creative brief that instructs the artist well enough for them to materialize the test variants. The artist can be a graphic designer, copywriter, animator, etc., and the brief should be adjusted in accordance with their types of work. Furthermore, ASO-related briefs should entail particular specifications to make sure the test assets meet Apple and Google’s requirements.


Step 5 – Produce:

The preparation phase should already determine what to create and how to create it. In production, the actual design or copywriting work may begin.

Step 6 – Test:

Real execution may begin as soon as test variants are produced. Execute the experiment by:

  • Picking the right testing method.
  • Choosing a suitable testing platform.
  • Deploying variants to the platform, following conditions of the chosen method.
  • Beginning the test.
Step 7 – Measure:

Compare the Control variant’s conversion rate (CVR) with the test variant’s to measure test results, then validate or invalidate the hypothesis, and finally make sense of the results – with critical reflection, not assumptions.

Step 8 – Handle the results:

There are 3 standard protocols to follow for the treatments of the results at this stage: Scale the variant/test, scrap it, or adjust it.

  • Scale a convincing win by expanding the variant to more languages, assets, etc., while keeping the same strategy. Scaling starts from production  – making a growth loop for CRO.
  • Scrap a loss that shows little to no value and is beyond salvation. Either revisit a previous step in the Preparation phase or restart the whole process and do the research again. This is a feedback loop that at least guarantees to learn in CRO.
  • Adjust anything in between, as long as it is possible and worthwhile to improve the variant. Change one detail at a time, develop a new hypothesis, then test again, and repeat. This is where the true optimization loop is made – with iterative testing.

If you would like to learn more about our frameworks or need help implementing it, don’t hesitate to contact us!

See you around!