Five Questions to Ask Before Hypothesis Testing

This question requires not an emotional response but a response backed by at least semi-serious research. And it’s not about having a paper analysis to cover yourself in case of failure and budget drain.

First, you need a plan. Preferably, you can make a real calendar plan (Excel will do), but a simple task list will also suffice. The main thing is that it should be clear:

• What tasks do we need to solve, and what exactly to do;
• Who and what we need to agree on;
• What materials and documents need to be found;
• What changes will we introduce into the processes;
• When will we do all this.

Imagine the project goal is to create a brand vlog. The plan with one line, "Start a channel on YouTube," immediately goes for a second round.

"1) Develop the style of the corporate page on YouTube. 2) Create visuals. 3) Create the page. 4) Announce to employees and ask them to subscribe."—can be used as a starting point.

"5) Compile a list of all available video materials. Responsible A.B., do by this date. 6) Select relevant materials among them, prepare descriptions in the required format, and upload them to the page. The person responsible is C.D., due by this date"—much better. Perhaps you should also think about a promotion plan.

It is a tricky question because the answer consists of two parts.

The first part is metrics. We need to understand what we are measuring and how. Traffic? Subscribers? Leads? Perhaps we are already measuring these indicators. They probably grow at some rate. Then, we are interested in the change in growth, not the indicator itself. We are interested in the growth change specifically related to our hypothesis. Therefore, we need to try to cut off external factors that affect changes in the indicator, such as the launch of a new product.

Here begins the second part of the answer to the question. You need to forecast changes in metrics “at rest. “Then, assume how the project or hypothesis will affect the dynamics of changing these values. That will be the goal.

Example: Traffic on the site this month is 10,000 visitors. Over the last year, it has grown by an average of 5% per month. (Question: What was the traffic a year ago? How many visitors will there be in 12 months? Answer at the end of the article.) Think about how many percent per month the traffic will grow if we launch the project. A “5%” answer would mean the project will not affect traffic. But if by 10%, our project will bring an additional 5% monthly traffic growth.

Finally, there should be at least two metrics. One strategic—for the sake of which we even started the project or launched the hypothesis testing. Say we want to get 100 more leads in six months than planned at the beginning of the year. The second goal is operational. It should show how quickly we are approaching the strategic goal. For example, if you need 100 additional leads from a new landing page, it would be logical to monitor traffic and conversion.

Next, we need to collect primary data on traffic and conversion and use simple math to estimate how they should grow in the remaining time to achieve the strategic goal. How realistic is it? Do we need to change the approach?

No: "Let's update the landing page. We’ll measure the traffic on it."

OK, but it could be better: "Update the landing page, on which traffic grew by 5% monthly. Within six months after the update, it will bring us 2,000 visitors with an average growth of +10% month over month."

The right way: "The updated landing page will bring 10,000 visitors in six months, about 1.2 thousand more than the original plan. The average increase should be +10% monthly, 5 percentage points faster than last year. The conversion should gradually increase from 2.0% to 2.6% due to constant UX improvement. This will give us 30 more leads than originally planned."

The most painful part of the exercise is calculating the budget and determining the resources needed. Ideally, agree on who and when will allocate people, money, equipment, or anything else necessary.

To quickly agree, you need to explain to the resource holders what the project or hypothesis is about. The worked-out, structured, and formatted answers to the previous questions will significantly help you. It will be much easier to explain the hypothesis’s essence to colleagues, show the approach’s seriousness, and transform from a requester into someone who offers opportunities.

The resource plan should be made as a schedule plan and discussed with the resource holders to determine whether they can allocate everything necessary at the right time and volume. If doubts arise, you should immediately consider alternatives and workarounds.

At this stage, you can use a unit approach. For example, we want to release 10 articles and adapt each for four channels. A copywriter spends four hours on one article, the editor—two, the illustrator two two, and the chief editor one one hour. Each rewrite requires two hours of a copywriter, one hour of an editor, and half an hour of an illustrator. Placing all article versions takes three hours of a content manager’s work. For 10 articles, it will be 10 times more—but over three months.

"We need $ 10k for advertising on Google."—no.

"We want to run ads on Google. We need $ 8k for search campaigns and $2k for the Display Network. The advertising manager will spend 20 hours on semantic search and initial campaign settings, then 2 hours a week for six months on regular optimization."—Good, but these are not all the resources.

"In addition to this, we will need a copywriter and editor resource for creating ads—20 hours of an external copywriter and 4 hours of our editor. We will outsource 15−20 illustrations for the Display Network to two designers. Outsourcing expenses—$4−5k; here is a table by months, and advertising expenses are also in a separate column. I have agreed with the heads of advertisers and our production that they will allocate resources at the right time. If necessary, we have two freelancers on low start who can be involved instead of employees."—A grown-up speaks.

It is an optional question. It makes sense to ask if the project is large, involves many departments, or is too unusual for the company. You need experienced employees with a broad business perspective to answer it.

The science is to calculate a couple of development models of the situation and understand at what points it is necessary to stop and check progress. The art is to discern hidden connections, predict the "butterfly effect," see the non-obvious impact of the project on other projects, and vice versa.

Here are some common negative scenarios:

• Cannibalization or freezing of revenue for the current or future periods;
• Profit reduction due to redirection of resources to new initiatives;
• Distracting potential customers from main products, slowing the sales cycle;
• Lowering the morale of the project team;
• Damage to the brand or reputation;
• Legal risks.

At the end of the day, everything might go wrong. That’s the point of hypothesis testing.

Well, yes.

And since you’ve read to the end, here’s the answer to my earlier question. Growth rounded down to the nearest whole.

The total traffic from January to December inclusive is just over 93,000 visitors.