Common Testing Mistakes and How to Avoid Them by Jibran Yousuf

Common Testing Mistakes and How to Avoid Them!

It has always been a point of concern for many marketers to find out the reason behind low conversion rates despite their efforts to bring targeted audience to the website or app. In this article, we will go through the common testing mistakes & how we can avoid them. Over the years Conversion Rate Optimization has gained a lot of importance as it helps to reduce customer churn and improve user experience thereby generating much needed revenue.

In the last blog, we learned about cditing & punching up the copy.

Even if we assume that your conversion rate is around 5%-10% that still means that the remaining 90% or 95% of your visitors are not converting! Surely that’s where you need to put your thinking caps on and think why did those 95% fail to convert even though you used highly targeted marketing to bring in relevant traffic that clearly had prior behavior or interest patterns suggesting that they would have been inclined to purchase your product or service.

Testing holds supreme importance in conversion rate optimization. In CRO, testing means that you need to identify the best possible version of your website. It ensures that the amount of effort you put in to bring in qualified traffic to the website converts into actual sales at a higher rate and brings in more revenue. Not only that, testing ensures that good customer experience leads to higher retention rates and hence increase CLV over time.

Importance of Correcting Mistakes:

The main point of testing is to identify flaws in your conversion funnel and take actions to correct them. It is also pertinent to point out that when doing CRO you need to have a buy in from all the stake holders most importantly, the tech team and your high-level management. All to often I have seen great CRO audits being done along with expertly crafted tests that propose the best versions of website landing pages only to be shot down by tech for not being included in their sprint planning or the boss thinks that more time could be spent on customer acquisition campaigns. So, if you are really going to go ahead with CRO then do it right and have the team buy-in.

Testing in CRO is a tricky business. Here are eight common testing mistakes and how you can avoid them.

Mistake#1 Not Letting A Test Run Long Enough:

Many marketers run their tests for a limited period of time and hence their test results are not statistically significant. In order to draw valid inferences from your tests, experts suggest that you should run test for at least a week. As Neil Patel puts it, CRO is a long-term game and you need to play by the book. You need strike a balance when it comes to the length of the test and how many conversions are good enough to draw reasonable conclusions.

Mistake#2 Testing Too Many Small Elements:

If you focus on testing minute features of landing features and not take a holistic approach to testing. Only then you are likely to draw the wrong conclusions from your test. Before going into testing, you need to draw a strategy where you identify elements that create a big impact, test them and then proceed to incremental changes.

Mistake#3 Just Testing Random Things:

Just don’t test random objects on the page. What you are testing needs to be backed by data e.g. a CTA button has achieved a high number of hovers but the click rate is no high thus requiring your intervention and testing various alternatives and finding the best version that gives you the right CTR. Before testing anything ask yourself, why do you want to test it and what should be the expected out come? Tests without a clear supporting hypothesis are bound to fail.

Mistake#4 False Positives:

In my experience I have learnt (mostly the hard way) that the number of iterations is directly proportional to false positive risks. One way to identify if you have a false positive on your hands is to test the winning iteration against itself and use back up tracking.

Mistake#5 Not Knowing When to Say When:

Running a test for far too log can also make it lose its validity. You need to cut out losing iterations quickly to focus on the ones creating significant results. Also, if no big improvements are coming in then you need to stop testing. Go back to the drawing board and re-strategize.

Mistake#6 Failing to Optimize for Each Traffic Source:

You need to test diverse traffic sources differently. For example, traffic coming from Desktop and Mobile behaves differently and your tests need to take this into account. A mobile user’s journey might entail discovery, awareness and creating buying intent but the same person could come to desktop to convert due to various reasons. Hence it is important to treat judiciously treat diverse traffic sources.

Mistake#7 Only Focusing on Conversion Rate:

Conversion rate without its proper context is useless. One of the most common mistakes while doing testing is focusing on final conversions and ignoring rest of the funnel. The point of testing is to ensure smooth user journey and that means optimizing all touch points in that funnel.

Mistake#8 Treating Low Traffic Websites the Same:

With low traffic sites, getting a statistically significant enough sample size is a real challenge. Employing the same tactics that would for high traffic sites when it comes to testing is a common mistake. One size fits all strategy rarely works. You need to outline the right time period and sample size for the test for it to produce valid inferences. Two ways to go around this problem are user testing and heuristic analysis of small groups. These are used to validate any changes or generate constructive feedback.

CRO tests bring high revenues when done correctly. If you can avoid these common mistakes your business can potentially end up saving a lot of money in these tough economic conditions.


    Introduction To Conversion Rate Optimization

    If we analyze the paradigm shifts of the Internet, we come across three paradigms in the past three decades, starting from the 1990s, with Yahoo & Lycos, to Google in the 2000s, followed by Social Media sites in 2010s.

    Photo by Leon Seibert on Unsplash

    And now the Internet is easily accessible by almost everyone, even in remote locations, which intrigued the businesses to grow their digital presence. And now every industry, including eCommerce stores, SaaS businesses, non-profit organizations, or even political campaigns, focus on their online presence. Now they all own a website, has their location on Google Map, takes online orders & sell their services, and influence the people around the globe right from their comfort of home.

    Analysis of customer behaviour shows that between 70–80% of people research a company online before visiting a small business or making a purchase with them. 97% of consumers go online to find a local store or local services.

    These paradigm shifts not only introduced a new scope of work but also added several new growth & marketing strategies & tactics to be implemented online for the hypergrowth of startups, & established businesses.

    In this article, we will discuss one of the essential processes in growth marketing, i.e. Conversion Rate Optimization or as we call it CRO.

    Conversion Rate Optimization (CRO)

    Before we dive deeper into the CRO, I would like to mention that the following content is based upon the learnings that I did during the First Week of Conversion Optimization, my nano degree course at CXL Institute, led by Peep Laja & team.

    This course’s lectures were conducted by Brian Massey, the Managing Partner at Conversion Sciences.

    Introduction To Conversion Rate Optimization

    When we talk about online marketing, there are two ways to increase your revenue. Either you start driving more traffic to your website to increase sales, or you improve the overall website effectiveness to increase the number of people who purchase something from you. Conversion Rate Optimization (CRO) focuses on the second way. Let’s get started.

    Conversion Rate Optimization

    Why CRO?

    Let’s consider that you have a website, and you are not getting enough conversion, visitors are not purchasing your products, your services are not selling, and you are not getting enough leads to convert them.

    There are two ways to tackle the current situation. Either design a new website landing page OR gather the data based on current visitors, & analyze it. Based on this analysis, update the chunks of website layout design. Which one sounds better? I am sure you would go for the latter. However, went for the first one, in November 2012, and updated the whole website layout.

    Finish Line Website on November 18, 2012
    Finish Line Website on November 19, 2012

    This step, damaged Finish Line business with an estimation of 3 Million US Dollars with a few weeks. This is where CRO comes into play, enabling you to do test-driven updates to your website, minimizing the risk factor of crashing the business.

    CRO primarily focuses on the following:

    • Decrease the Cost per Acquisition (CAC)
    • Increase the number of leads
    • Get more sales

    So If your business has fewer leads, high bounce rate with a high cost of acquisition, you must focus on CRO processes to grow your business exponentially without risking the current growth.

    The Basics

    When we talk about improving the Conversion Rate, try sharing this objective with the colleagues around you. You will be bombarded with ideas, from your team, your boss, or the ideas that you took as inspiration from your competitors.

    And most of the time, these ideas make their way onto the live website, without a proper framework to validate all those ideas. This happens due to several reasons, sometimes due to the political will of your superiors, while sometimes due to your confirmation bias of the idea that you suggested.

    How to avoid this problem? Most of the people would say, let’s learn how to manage these ideas. No, the human brain has the built-in ability to distinguish ideas. We use this ability in our daily lives. Now, remember how many times you analyzed the current situation and curated a perfect status update or tweet for your social media profile, that brought a lot of engagement?

    The last time you bought a product from Amazon, you got stuck between two variants, and you quickly went into reviews to decide which one to buy? All of these are the example that the CRO mindset is already wired into our brain; we just need to use in our business.

    Understanding Conversion Rate Optimization
    Photo by Raquel Martínez on Unsplash

    The above examples are also the daily life example of A/B testing. As you compare two updates, analyze the data, and choose the one that benefits you at most.

    Now that you have an idea of what CRO is, we will talk about the three founding pillars:

    • Managing the ideas
    • Increasing the sample size
    • Improving the quality of the sample.

    Managing Ideas

    Managing Ideas for Conversion Rate Optimization
    Photo by Aaron Burden on Unsplash

    Ideas are the oxygen for a digital marketer, that is large in number. As I said before, share the objective with your team, and you will be quickly having 10–20 ideas within an hour. Our brain is programmed to have multiple ideas. So how to decide which one is worth trying? And how to politely decline the idea of your boss or any senior authority in the business?

    Simply, create a spreadsheet, and write down all the ideas in that list, gather it from the team & even from your boss.

    Writing A Hypothesis

    Now, write a hypothesis for every idea, in the given format.

    The hypothesis to enhance ideation

    Taking an example of Nonfig, I had the idea to change the text on the CTA button in your lead opt-in form on your main landing page, so I wrote the hypothesis for that.

    If I update the CTA button text of opt-in form to Claim Your Free 3 Months
    I expect to get more leads.
    As measured by Number of leads in Google Analytics

    Ranking Ideas

    Once you are done with the hypothesis, rank your ideas based upon these three factors, usually rating them from 1 to 5.

    • Impact — the effect of the idea post-implementation
    • Confidence or Proof — the probability of achieving the desired impact
    • Effort to Test — the amount of work required to conduct this test

    Updating the CTA button text can create a significant impact, has the credibility to succeed based on the Copytesting rules, and the effort to test this change is minimal. Whereas changing the graphical assets used on the landing page from photos to vector might create a huge impact too. But the effort to test is high, and therefore we know the first test can be easily prioritized.

    Sources Of Insight

    Whenever we talk about initiating a test, it’s recommended to analyze the ideas based upon a number of data sources.

    First of all, Google it. In this continually changing world, a number of marketers are regularly running tests and putting their findings on the web. Go through them, and check if your ideas are even testable or not.

    Find out what is required to execute these ideas, maybe some just need a simple fix, whereas some need an update in layout design.

    Gather Insights for Conversion Rate Optimization
    Photo by Stephen Phillips – on Unsplash

    One of the most recommended sources of data is your website analytics, and it has everything that you are looking for before running a test. Analytics can help you save hours of work, and you will be able to understand the visitor behaviour more accurately analyzing the patterns.

    Apart from the above, many tools give you insights and save you from the disappointment of running a test that shouldn’t be run in the first place. A few of these tools are:

    • Hotjar — provides feedback through tools such as heatmaps, session recordings, and surveys.
    • SessionCam — provides a click heatmap, a scroll map, and a screen recording feature to view how users behave on pages.
    • Tobii — focuses on sensor technology that makes it possible for a computer or other device to know where a person is looking.

    A/B Testing Basics

    A/B Testing in Conversion Rate Optimization

    Whenever a person listens to the term Conversion Rate Optimization, their mind instantly thinks of A/B Testing. However, CRO is much more than just A/B Testing.

    A/B Testing is the best way to collect visitors’ behavioural data by presenting them with different versions of a creative, or layout, or even a copy on your website.

    When we talk about getting feedback about a particular feature, or an update, we consult it with our team, our friends, & the family circle. Most of the time, the feedback is not accurate, and they act as pretenders & liars.

    A/B testing allows you to run the test, with both ends. As they both ends do not know the actual person behind the insight that you receive. CRO specialists will only get the numbers in forms of analytics. Whereas the visitor won’t even know that they are being tested, so their response is natural.

    The key to running A/B Testing is to understand which tests need to be run. This is vitally important to avoid running tests where they are not required. Also, keep in mind that it’s not necessary that the test variant that you are supporting only because of your own confirmation bias will lead the test.

    Bring It All Together – Conversion Rate Optimization

    Conversion Rate Optimization is one of the fundamental processes of Growth Marketing. We learned what CRO is & why is it important. We also shed light on how great ideas are lost, while ideas supported by political will make it to the top. However, we came across the framework to manage ideas. Insights are essential. Once you have the list of sources of raw data, combined with the powerful features of excellent tools, it becomes the perfect recipe to run A/B Tests.

    In the next article, I have discussed the Best Practices for the CRO.

    Do you have any questions? Feel free to ask in the comments section below.