Five Questions to Ask Before Hypothesis Testing

Published on 15 April 2024

And before launching any projects that might take more than a week

The brainstorming session is over, the crackers are eaten, ideas are jotted down, and responsible employees are assigned. It’s ready to kick off. But before diving into testing or project work, spending budget and colleagues' time, I ask those responsible to describe the hypothesis and prepare answers to five questions:

  1. Why do we need to do this?
  2. How exactly will we do it?
  3. How will we evaluate the result?
  4. What resources will we need, and where will we get them?
  5. What could go wrong?
“The brainstorming session is over, the crackers are eaten“

Why Do We Need to Do This?

This question requires not an emotional response but a response backed by at least semi-serious research. And it’s not about having a paper analysis to cover yourself in case of failure and budget drain.
Exploring even the most familiar problems can lead even the most seasoned marketers to discover new ideas, opinions, and fresh input.
If the marketer is savvy, that’s when the magic happens. External enrichment transforms an idea into a conscious project with a clear goal, scope, prospects, evaluation methodology, and a schedule-resource plan.

Won’t cut it: "We want to increase our presence on Facebook and Instagram because they have a large audience and everyone loves reels."

Better: "We want to advertise on Facebook. It has over three billion users, more than 30% aged 25 to 34 (our target audience). The CTR in the Industrial & Commercial category (also our target audience) is about 1%, and the average cost per click is $0.8."

Much better: "We want to develop a corporate page, advertise, and grow a community on Facebook. Our target audience is there (see above). Between 55% to 75% of B2B companies use social networks to evaluate suppliers. With Facebook, we can reduce the average warm-up time of our leads from the first touch to MQL from 6 to 4 months (see internal analytics)."

How Exactly Will We Do This?

First, you need a plan. Preferably, you can make a real calendar plan (Excel will do), but a simple task list will also suffice. The main thing is that it should be clear:
  • What tasks do we need to solve, and what exactly to do;
  • Who and what we need to agree on;
  • What materials and documents need to be found;
  • What changes will we introduce into the processes;
  • When will we do all this.
Important: The plan should define the project’s scope and required resources.
Imagine the project goal is to create a brand vlog. The plan with one line, "Start a channel on YouTube," immediately goes for a second round.

"1) Develop the style of the corporate page on YouTube. 2) Create visuals. 3) Create the page. 4) Announce to employees and ask them to subscribe."—can be used as a starting point.

"5) Compile a list of all available video materials. Responsible A.B., do by this date. 6) Select relevant materials among them, prepare descriptions in the required format, and upload them to the page. The person responsible is C.D., due by this date"—much better. Perhaps you should also think about a promotion plan.

How Will We Evaluate the Result?

It is a tricky question because the answer consists of two parts.

The first part is metrics. We need to understand what we are measuring and how. Traffic? Subscribers? Leads? Perhaps we are already measuring these indicators. They probably grow at some rate. Then, we are interested in the change in growth, not the indicator itself. We are interested in the growth change specifically related to our hypothesis. Therefore, we need to try to cut off external factors that affect changes in the indicator, such as the launch of a new product.
We need to calculate what addition our hypothesis will contribute to the current growth rate.
Here begins the second part of the answer to the question. You need to forecast changes in metrics “at rest. “Then, assume how the project or hypothesis will affect the dynamics of changing these values. That will be the goal.

Example: Traffic on the site this month is 10,000 visitors. Over the last year, it has grown by an average of 5% per month. (Question: What was the traffic a year ago? How many visitors will there be in 12 months? Answer at the end of the article.) Think about how many percent per month the traffic will grow if we launch the project. A “5%” answer would mean the project will not affect traffic. But if by 10%, our project will bring an additional 5% monthly traffic growth.
“Goal Setting“
Finally, there should be at least two metrics. One strategic—for the sake of which we even started the project or launched the hypothesis testing. Say we want to get 100 more leads in six months than planned at the beginning of the year. The second goal is operational. It should show how quickly we are approaching the strategic goal. For example, if you need 100 additional leads from a new landing page, it would be logical to monitor traffic and conversion.

Next, we need to collect primary data on traffic and conversion and use simple math to estimate how they should grow in the remaining time to achieve the strategic goal. How realistic is it? Do we need to change the approach?

No: "Let's update the landing page. We’ll measure the traffic on it."

OK, but it could be better: "Update the landing page, on which traffic grew by 5% monthly. Within six months after the update, it will bring us 2,000 visitors with an average growth of +10% month over month."

The right way: "The updated landing page will bring 10,000 visitors in six months, about 1.2 thousand more than the original plan. The average increase should be +10% monthly, 5 percentage points faster than last year. The conversion should gradually increase from 2.0% to 2.6% due to constant UX improvement. This will give us 30 more leads than originally planned."
Laid-out goals almost always mean knowing exactly how we will achieve the result.

What Resources Will We Need, and Where Will We Get Them?

The most painful part of the exercise is calculating the budget and determining the resources needed. Ideally, agree on who and when will allocate people, money, equipment, or anything else necessary.

To quickly agree, you need to explain to the resource holders what the project or hypothesis is about. The worked-out, structured, and formatted answers to the previous questions will significantly help you. It will be much easier to explain the hypothesis’s essence to colleagues, show the approach’s seriousness, and transform from a requester into someone who offers opportunities.
During the discussion of resources, remember that your project will be stretched over time. Resources will be needed not immediately but according to the schedule. This can be a crucial argument for the decision-maker.
The resource plan should be made as a schedule plan and discussed with the resource holders to determine whether they can allocate everything necessary at the right time and volume. If doubts arise, you should immediately consider alternatives and workarounds.

At this stage, you can use a unit approach. For example, we want to release 10 articles and adapt each for four channels. A copywriter spends four hours on one article, the editor—two, the illustrator two two, and the chief editor one one hour. Each rewrite requires two hours of a copywriter, one hour of an editor, and half an hour of an illustrator. Placing all article versions takes three hours of a content manager’s work. For 10 articles, it will be 10 times more—but over three months.

"We need $ 10k for advertising on Google."—no.

"We want to run ads on Google. We need $ 8k for search campaigns and $2k for the Display Network. The advertising manager will spend 20 hours on semantic search and initial campaign settings, then 2 hours a week for six months on regular optimization."—Good, but these are not all the resources.

"In addition to this, we will need a copywriter and editor resource for creating ads—20 hours of an external copywriter and 4 hours of our editor. We will outsource 15−20 illustrations for the Display Network to two designers. Outsourcing expenses—$4−5k; here is a table by months, and advertising expenses are also in a separate column. I have agreed with the heads of advertisers and our production that they will allocate resources at the right time. If necessary, we have two freelancers on low start who can be involved instead of employees."—A grown-up speaks.

What Could Go Wrong?

It is an optional question. It makes sense to ask if the project is large, involves many departments, or is too unusual for the company. You need experienced employees with a broad business perspective to answer it.
“Something Went Wrong“
The science is to calculate a couple of development models of the situation and understand at what points it is necessary to stop and check progress. The art is to discern hidden connections, predict the "butterfly effect," see the non-obvious impact of the project on other projects, and vice versa.

Here are some common negative scenarios:
  • Cannibalization or freezing of revenue for the current or future periods;
  • Profit reduction due to redirection of resources to new initiatives;
  • Distracting potential customers from main products, slowing the sales cycle;
  • Lowering the morale of the project team;
  • Damage to the brand or reputation;
  • Legal risks.

At the end of the day, everything might go wrong. That’s the point of hypothesis testing.

But It’s Almost SMART!

Well, yes.

And since you’ve read to the end, here’s the answer to my earlier question. Growth rounded down to the nearest whole.
The total traffic from January to December inclusive is just over 93,000 visitors.
New articles in your mailbox
No more than once a month.
You agree to our Privacy Policy