I wrote the below chapter for TriKo's (Now Kromatic's) The Real Start-Up Book. 

How to Write a Hypothesis

Lean startup practices turn project managers, business leaders and designers into scientists who constantly validate their ideas through running an array of experiments. But  experiments can get out of hand and turn perfectly sane people into mad scientists. A sure way to keep one’s sanity is to start the experiment with a strong hypothesis.

A strong hypothesis will give structure to the experiment. It will tell you what you are testing and what you expect to get out of the test. By stating the expectations, you are stating the goals that the experiment has to hit to make it a success or failure. This will help you define when to determine that the experiment needs to be scrapped or if it is ready to be taken to market.

Key elements of writing a good hypothesis:

  1. The change that you are making

  2. The aspect that will change

  3. The success or fail metric

  4. How long are you going to run the test

A hypothesis will end up looking like this:

This new feature (The Change) will cause an 10% increase (The Metric) of new users visiting the homepage (The Impact) in 3 months (The Timeframe).

Let’s break down each aspect:

The change: This is the aspect that you are going to change, launch or create that is going to affect your business or product overall. It can be as simple as changing the color of a button or as big as launching a new marketing campaign. Make sure that only one aspect is changed at a time, otherwise there is no way to tell which aspect contributed to the effect.

The impact: This is the expected results of the experiment. If you change x, then you expect y to happen.

The metric: This is a measurement that needs to be hit or surpassed. This can be a fail metric where if the experiment does not meet the minimum goal, then the project must completely pivot into a new direction. The metric can also be a success metric, where the experiment is deemed to be a success it it hits the goal. Choosing between success and fail metric is dependent on if you want the baseline to know when to scrap a project or when to launch a project.

The timebox: This is the length of time it takes to run the test. If the timebox is too short then the data size might be too small or the effects might not have had time to take place. But if the timebox is too long then you are wasting valuable time collecting unnecessary data.

Let’s go through an example scenario of writing a hypothesis:

Say that you are a product manager at startup that creates a mobile app to help waiters and waitresses keep track of their tips. You have noticed that users who document their tips 4 times a week have a higher retention rate. You want to see if you can increase the number of times current users use the app within a week.

What are you going to change within the app?

How about adding a notification system so the user can set reminders to ping them at the end of a shift.

What do you want the outcome to be?

You want more users to open the app 4 or more times in a week.

What is the metric of failure or success?

At this time you have 50,000 monthly users and 10,000 use the app 4 days a week. You want to increase the current user’s rate of opening the app from 10,000 to 15,000. This translates to a 10% increase.

How long are you going to run the test?

This always depends on a number of variables within the company but let’s say that you are at a midsize company that has a little more time to get the correct data. So let’s say three months.

The end hypothesis would be: If I add a notification feature that allows the waiter/waitress to set reminders to add in his/her tips, then I am going to see a 10% increase in number of users opening the app 4 times or more in a week over the next three months.   

Below is a worksheet that will test if you can figure out the strongest hypothesis for a given scenario.

Scenario 1: You work for a company that rents out toddlers’ clothes. It is a monthly subscription where families get a box of 5 pieces of clothing and when the toddler grows out of them, they return the clothes for a new box. The data shows that there might be a correlation between members who frequently send items back to higher customer retention rates. Your goal is to have members return more boxes. You have decided that you can do this by adding pieces that are seasonal, holiday or super trendy so that the family will need to keep updating the clothes.  

  1. By adding one piece of special occasion clothing, you will see a 10% rise in returned boxes in 3 months.

  2. If you include one special occasion outfit, a new designer piece and a seasonal accessory, then you will see a 15% increase in returned boxes in the next 12 weeks.

  3. When you add three seasonal pieces then families will learn to request more items and you will see growth in the next 2 months.

  4. By including one trending designer piece, then you will see a 15% increase in requests for those designers once the experiment is completed.

 

Scenario 2: You already made your millions with the Uber for parrots so you decided to invest your money into saving the manatees. You designed a tracking app that shows boaters where herds of manatees are sleeping so they don’t run the herds over. You are having a hard time getting the boaters to download the app so you decided to start advertising. You want to conduct a test to see if a promotion will increase the app’s downloads.  

  1. If you pair up piers to give 10% off of a month of docking their boat if the owner download the app, then you will see a 10% increase in downloads over the next three years.

  2. If you give out 10% coupons to boat rentals for downloading the app, 15% off tack shops and advertise around piers, then you will see an increase of 15% new downloads the next 3 months.

  3. By pairing up with 10 boat rentals to give a coupon of 10% off the boat rentals for downloading the app, you will see a 5% increase of downloads over a 6 month period.  

  4. When you have a special where someone downloads the app, they get a one of a kind lure at Ted’s tack shop (which has 15 stores in Florida), then you will see 10% decrease in Manatee deaths over the next 5 months.


Scenario 3: Your Labrador is obsessed with a tennis ball and you are tired of throwing the slobbery thing. It inspired you to start a drone company that drops tennis balls and takes funny pictures of the dogs. Your customer support team has received complaints that it is hard to understand how to download the pictures from iphone app. You want to test moving the photos section to various parts of the app.

  1. If you add a photos section to the navigation bar, then you will see a 5% increase of new users over a 4 month period.

  2. If you advertise about the photo feature in your app, you will have more users and fewer complaints within the next 10 weeks.

  3. If you add three pages to the onboarding process that explains the pictures, then you will get 25% increase of dog pictures.

  4. By moving the download photos to be part of the home screen, you will receive 50% fewer complaints about the photos section in the next three months.

 

Scenario 4: You are so sick of wearing the same outfits that you developed an AI software to pick out your clothes every morning. A venture capitalist saw your tweets about it and gave you a million dollars to start the company. You need the AI to address weather conditions when choosing the clothes. You want to run an experiment to test a method for collecting data for when it is 80 degrees and sunny.

  1. If you poll people in popular cities on sunny days, then you will be able to add 5% more data points.   

  2. If you see what is in clothing stores on sunny days, you will be able to add 10% more data points to the algorithm in a month.

  3. If you send out a survey to ask people what they are wearing when it is 80 degrees and sunny, then you will get a 75% answer rate in a week.   

  4. By collecting data points of Instagram selfies dates to days of 80 degrees and sunny, your AI can identify 75% of the clothing in 3 months.

 

Scenario 5: Men’s socks are a great way to jazz up an outfit so you decided to start a men’s sock e-commerce store. Your customers are not completing the check-out process and usability tests show that some users question the “site’s security”. You want to add a small adjustment to the payments page to see if more users complete the check-out process.

  1. If you add a password strength indicator then more people will create passwords in the next 2 months.

  2. If you add a lock icon next to the credit card info, the completion of the checkout process will increase by 15% in 3 months.

  3. If you make the site prettier, the completion of the checkout process will increase by 25% in 6 months.

  4. If you add a review page before the confirmation page, then 20% of customers will be able to complete the checkout flow in 10 mins.

 

Answers:

    1 - A

    2 - C

    3 - D

    4 - D

    5 - B

 

Learn from your mistakes. Look at the questions that you got wrong and pay attention see which key element is either missing or vague.

 

Common Mistakes:

  • There are too many variables. If you are testing multiple things then you can not pinpoint which variable caused the results.

  • There has to be an achievable metric attached to the hypothesis. You have to know the point at which experiment succeeds or fails.

  • The success or failure directly is not linked to the experiment. If the success or failure could have happened because of a number of variables, then you don’t know if the experiment was the reason for the change.

  • The time box is too long or too short. Some experiments are going to take a longer time but you have to make sure that the time box is reasonable for the growth rate of your company. You don’t want to have an experiment that takes years or that is over extended past the runway of your company.

 

Other Resources