Formulation Of Good Hypothesis Statements

Hypothesis is defined as “A tentative assumption made in order to draw out and test its logical or empirical consequences.” When it comes to landing page optimization, the test hypothesis is the assumption you want to test against what you already have. In order to increase the probability of success of your A/B tests, you need to form a strong hypothesis through researching and observation.

Expert Marketers don’t go about testing random ideas hoping to stumble upon a working solution. Instead, they use solid test hypothesis.

A hypothesis provides a solid and targeted A/B testing roadmap that will generate lifts and provide vital lessons. Thus, every test should start with a hypothesis that you are trying to prove or disapprove. So formulating a hypothesis is the quicker, more accurate and less expensive way of solving a problem.

Here are 7 steps to take to formulate a strong A/B testing hypothesis

1. Define your problem

Defining your problem is the first thing that needs to be done. What is it that you want to test or solve? Is it to double your sales or to increase the number of opt-ins? Whatever your goals are, they need to be clearly defined, quantifiable, and measurable. This should give you a clear idea of what your new design should solve including the process that will be followed to achieve the results.

2. Find out the reasons behind the numbers

Now that you have defined your problem and you have a clear picture of what it is you want to achieve, the next thing that follows is an in-depth analysis of the current problem. This can be equated to sharpening the ax. Basically, you want to take as much time as possible to learn the reasons behind your numbers.

You won’t be able to form an accurate hypothesis without studying what is happening in the website where you want to test your A/B test. Now that you are already looking for better variables to improve your conversion rates, it is only logical that you find the reasons that brought you to this current situation. Why are you experiencing high bounce rate? Why aren’t you seeing more conversions? Why are most of your customers failing to complete the payment process? These are obviously some of the reasons that may push you to improve your website.

The only way to discover areas of improvement in your website is to study your target market. You also need to get customer feedback through comments, social media, surveys and email.

3. Talk to your visitors

It is important to get real feedback from your visitors. One way is to use surveys—both entry surveys and exit surveys that are used to discover your visitor’s objectives and determine whether their goals have been met respectively this is aimed at understanding what they want or what their desires are.

Knowing the reasons behind their decisions and actions is the most important part of the survey. Therefore, do not hesitate to ask them to give reasons for their actions in the survey. For instance, you can place an exit survey at the end of a buying process to ask them why they bought your product. You could also place an exit survey immediately they abandoning a buying process to understand why they did so.

You could also use analytics tools to gather quantitative data such as location, devices used, bounce rates and number of visitors and so on. In other words, both surveys and analytics tools can complement each other when it comes to gathering information about the customer.

4. Use segmentation to get actionable data

In statistics, averages don’t tell you the whole story. Segments do, and that is why segmentation is an important step in the formulation of hypothesis. For example, an experiment may show that a certain product is not performing well, but upon further analysis, it may be discovered that majority of people who buy the product are women aged between 18 and 29 years. Upon further investigation, it may turn out that ads for the product were being targeted to the general population.  So when you do segmentation, it may eventually occur to you that you should concentrate your marketing efforts on the women who fall in the 18-29 age bracket. In other words, segmentation gives you actionable data, which would otherwise be useless without it. There are many approaches to segmentation. Examples include:

Source segmentation: involve separating the visitors who come to your site based on their sources e.g desktop or mobile, android or iOS, email or social media. Find all the metrics associated with all these segments such as bounce rates, number of visitors, conversion rates and regions.

Behavioral segmentation : See what elements your customers focus most in your website. These days there are eye-tracking technologies that make it possible to study the elements that attract the most attention. Find out who are the 20% of your segment that bring the 80% of revenue (refer to Pareto principle). Another way one can gather data is through usability testing where one observes the behaviors of customers and records the observations.

Outcome segmentation: here, you focus on different types of products that have been purchased, the number of people who did not complete purchase, where most of your orders are coming from etc.

In short, the aim of segmentation is to find out where your most profitable segment comes from and focus your efforts on that segment.

5. Articulate a Hypothesis for your test

Now that you have gathered enough evidence to show what or where the problems is, it is time to state why you think the problem occurs.

Your hypothesis should have the following characteristics:

  • It is goal oriented—it clearly states what needs to be accomplished
  • It can be tested—it can easily be implemented
  • It is insightful—looking at the hypothesis, one should learn something about the problem.

An example of hypothesis

Problem: less than 5% of visitors buy the mobile app

Hypothesis: The text in the CTA button does not provide a clear message to the customer. The text needs to be changed from “Get it” to “Download you app now”.

6. Test substantial variations based on your Hypothesis

We can call this the brainstorming stage. After determining the problem and articulating a hypothesis. The next thing that follows is coming up with substantial variations based on your hypothesis.  Taking the above example, the hypothesis states that “The text in the CTA button does not provide a clear message to the customer.” The substantial variations could include things like changing the color of the button, changing position of the CTA on the landing page, changing the wordings, creating different icon etc. The substantial variations of your hypothesis are meant to bring you closer to the solution as quickly as possible and provide you with insights.

7. Analyze results to validate your hypothesis and Repeat

Once you’ve managed to articulate your hypotheses and test substantial variations, it’s time to analyze results to validate your hypothesis. You need to have sufficient test results in order to analyze and compare. When you are analyzing your tests with the aim of implementing solutions, you should bear in mind that revenue is the ultimate measurement of improvement. Customer feedback and analytics are tools you can use. You should look at the data your customers have left to help you choose the elements that need to be analyzed. The various elements you could test include:

  • CTAs—colors, texts, size
  • Images—placement size
  • Headlines—size, length, style, tone text color
  • Testimonials—placement, number, length
  • Videos—number, with or without videos
  • Forms—files type, color, number of fields
  • Shopping cart—icon, text, number of steps
  • Copywriting—long text or short, style, tone

After learning from your results, you should start the process all over because there is always room for improvement. In marketing, you never hit the perfect solution and even there is no way to know that you have reached perfection. Constant improvement is the name of the game in this field. So, it is a continuous process.

About the Author:

Steven Sanchez has been actively involved in SEO and Internet marketing since 1999. Steven’s knowledge and experience have made him one of the most respected and referenced SEO’s in the industry and his passion for innovation and growth has led his company, Internet Marketing Invesp, to become one of the world’s leading online marketing firms.



Want to know the secret to always running successful tests?

The answer is to formulate a hypothesis.

Now when I say it’s always successful, I’m not talking about always increasing your Key Performance Indicator (KPI). You can “lose” a test, but still be successful.

That sounds like an oxymoron, but it’s not. If you set up your test strategically, even if the test decreases your KPI, you gain a learning, which is a success! And, if you win, you simultaneously achieve a lift and a learning. Double win!

The way you ensure you have a strategic test that will produce a learning is by centering it around a strong hypothesis.

So, what is a hypothesis?

By definition, a hypothesis is a proposed statement made on the basis of limited evidence that can be proved or disproved and is used as a starting point for further investigation.

Let’s break that down:

It is a proposed statement.

  • A hypothesis is not fact, and should not be argued as right or wrong until it is tested and proven one way or the other.

It is made on the basis of limited (but hopefully some) evidence.

  • Your hypothesis should be informed by as much knowledge as you have. This should include data that you have gathered, any research you have done, and the analysis of the current problems you have performed.

It can be proved or disproved.

  • A hypothesis pretty much says, “I think by making this change, it will cause this effect.” So, based on your results, you should be able to say “this is true” or “this is false.”

It is used as a starting point for further investigation.

  • The key word here is starting point. Your hypothesis should be formed and agreed upon before you make any wireframes or designs as it is what guides the design of your test. It helps you focus on what elements to change, how to change them, and which to leave alone.

 

How do I write a hypothesis?

The structure of your basic hypothesis follows a CHANGE: EFFECT framework.

 

While this is a truly scientific and testable template, it is very open-ended. Even though this hypothesis, “Changing an English headline into a Spanish headline will increase clickthrough rate,” is perfectly valid and testable, if your visitors are English-speaking, it probably doesn’t make much sense.

So now the question is …

 

How do I write a GOOD hypothesis?

To quote my boss Tony Doty, “This isn’t Mad Libs.”

We can’t just start plugging in nouns and verbs and conclude that we have a good hypothesis. Your hypothesis needs to be backed by a strategy. And, your strategy needs to be rooted in a solution to a problem.

So, a more complete version of the above template would be something like this:

 

In order to have a good hypothesis, you don’t necessarily have to follow this exact sentence structure, as long as it is centered around three main things:

  1. Presumed problem
  2. Proposed solution
  3. Anticipated result

Presumed problem

After you’ve completed your analysis and research, identify the problem that you will address. While we need to be very clear about what we think the problem is, you should leave it out of the hypothesis since it is harder to prove or disprove. You may want to come up with both a problem statement and a hypothesis.

For example:

Problem Statement: “The lead generation form is too long, causing unnecessary friction.”

Hypothesis: “By changing the amount of form fields from 20 to 10, we will increase number of leads.”

 

Proposed solution

When you are thinking about the solution you want to implement, you need to think about the psychology of the customer. What psychological impact is your proposed problem causing in the mind of the customer?

For example, if your proposed problem is “There is a lack of clarity in the sign-up process,” the psychological impact may be that the user is confused.

Now think about what solution is going to address the problem in the customer’s mind. If they are confused, we need to explain something better, or provide them with more information. For this example, we will say our proposed solution is to “Add a progress bar to the sign-up process.”  This leads straight into the anticipated result.

 

Anticipated result

If we reduce the confusion in the visitor’s mind (psychological impact) by adding the progress bar, what do we foresee to be the result? We are anticipating that it would be more people completing the sign-up process. Your proposed solution and your KPI need to be directly correlated.

Note: Some people will include the psychological impact in their hypothesis. This isn’t necessarily wrong, but we do have to be careful with assumptions. If we say that the effect will be “Reduced confusion and therefore increase in conversion rate,” we are assuming the reduced confusion is what made the impact. While this may be correct, it is not measureable and it is hard to prove or disprove.

To summarize, your hypothesis should follow a structure of: “If I change this, it will have this effect,” but should always be informed by an analysis of the problems and rooted in the solution you deemed appropriate.

 

Related Resources:

A/B Testing 101: How to get real results from optimization

The True Value of Data

15 Years of Marketing Research in 11 Minutes

Marketing Analytics: 6 simple steps for interpreting your data

Website A/B Testing: 4 tips to beat an unbeatable landing page

0 comments

Leave a Reply

Your email address will not be published. Required fields are marked *