❓How do I do product experiments? Step by step

(This is an answer to a question from my previous post, check the link below)

Product experiments are about driving users to do a desired behavior.

The experiment starts with the Outcome I´m focused on. Usually a business lever, like increasing NMRR (New Monthly Recurrent Revenue).

Then I break down the Outcome math: NMRR = N° of New Subscribers * Price.

Then I prioritize the variable I have control over: the Price.
Other variables I´ll work on soon.

Then I define the Output by working backward:
Question: Which behavior should a user do immediately before the NMRR increase?
Answer: Pay the 1st Subscription.

Then I define the Input.
Question: When do users have access to the variable I´m changing?
Answer: When they access the Paywall.

So we have:
Outcome: NMRR
Output: N° of Users who Paid the 1st Subscription.
Input: N° of Users who Accessed the Paywall

Changing a variable can impact a lot of things. But the Outcome and Output are the most important in the experiment's 1st cycle.

With that, evaluating the experiment result gets easy using these rules of thumb:

Success Criteria:
A) Challenger´s Outcome Rate should be greater than the Champion's.
B) Challenger´s Output Rate should be greater than the Champion´s.

The Champion is the current variable (Price). The Challenger is the new variable (New Price).

Outcome Rate = Outcome / Input (NMRR per User who Accessed the Paywall)
Output Rate = Output / Input (Subscription Payment Conversion Rate per User who Accessed the Paywall )

Intended Actions:
- If A & B are true: move to the next business lever.
- Else if A is false: Evaluate if it is worth experimenting with a new hypothesis (like a less expensive price) or moving to the next variable)
- Else if B is false: We moved the Outcome needle, but we have room to improve it by experimenting with increasing the Output. Then I start an experiment where the Outcome becomes N° of Users who Paid the 1st Subscription, repeating this process.

According to your needs, the Success Criteria will need statistical significance or just practical significance. But this is a subject for later.

To get the context of the previous post, check this:https://www.linkedin.com/feed/update/urn:li:activity:7078015597875200000/

Work smart. Keep growing 🚀