When NOT to Test: 5 Conditions That Make Website Testing a Bad Idea

Posted by SiteTuners on 17 December 2014 | Comments


Testing is good in online marketing. However, it isn’t everything - remember what testing is and what it isn’t.

Testing is a way to validate the ideas for conversion rate improvement that you deploy on your site. That’s it. Testing just tells you if you've made a good decision, but it doesn't give you the ideas.

Aside from not treating website testing as if it’s synonymous to conversion rate optimization (CRO), here are 5 situations when you shouldn't test:

1. When problems are obvious

Forget testing and just fix the glaring problems on your site. While it’s true that you should test most important things, you should still consider best practices on effective copywriting, making the call-to-action clearer, and removing visual distractions.

If best practices worked 9 out of 10 times, that’s probably higher than the success rate for most of your tests.

2. When problems are site-wide (not isolated)

Testing individual pages is okay, but don’t forget to look at the whole experience.

If you create alternative versions of your product detail pages in an e-commerce website, for instance, that are way better than what they replaced, the problem will be the rest of the site experience. You might have that horrid homepage with sliding banner ads, your products might be impossible to find, or your checkout process usability might be poor.

Your site's performance will suffer because of the contrast between the part that you fixed and the context around it that still leaves a lot to be desired.

A lot of times, things can't be fixed by tactical testing because they’re not isolated to one part of your site - they’re part of the larger experience.

3. When there is not enough traffic

Even with basic A/B testing, if you don’t have 10 conversion actions a day, you shouldn't be doing tests.

You can try to compensate for the data rate and measure different things. For example, the conversion action in an e-commerce scenario might not be an actual sale- it might be adds to cart. If you know, for instance, that 1/3 of things added to the cart are eventually bought (so you have 3 times as many adds to cart as you do sales), you can measure that if you're fixing the upstream part of the experience.

The conversion action might also be just a navigation click. If you have the conversion action deeper in your site, the micro-conversion of a click in the right direction might be a good tactical independent thing that you can measure and test for.

4. When traffic is changing, seasonal, or spiky

If you can sample people from a certain population or group and figure out how they behave, then you’ll be able to predict the behavior of future visitors to your site.

The problem with the world of online marketing is it's not clean at all. It's difficult to sample from the same population - you have competitors, seasonality, and changes to your traffic mix. And anything fundamental that you change is going to affect the test, and you no longer get that purity of predicting how a certain group of people acts.

For example, if you have a tax offer, the people taking on the offer 6 months in advance are not the same as the ones ordering it a week before the tax deadline. For flower websites, customers care about urgent delivery the day before Mother’s Day. They don’t care about price like the people who plan ahead. In travel, you have seasonality with people from cold places coming to warm places in the winter, and vice-versa.

These are all different populations.

So, if you have strong seasonality or changing demographics or behaviors coming to your site, you can’t extrapolate that. If you do a test on a flower site a week before Mother’s day, you can only use that information for that same group of people. You can’t use it in off-season because people will behave differently.

If you know that there’s some seasonality or a holiday when people act weird, quarantine that part of the data out when you’re doing your analysis.

5. When you are ignorant of basic math

Some people ignore the math and the statistics altogether and just try to lock in the winner right away. You really can’t get proper statistics without collecting enough samples. If you’re working with too little data, cutting tests off early, or not testing in full week increments, you’re not doing math. You might as well be flipping a coin.

Keep Testing in Proper Perspective

Testing has its place in online marketing, but it’s not CRO. And there are situations when testing will do more harm than good. If the problem is conspicuous or site-wide, or if you don’t have the data rate or traffic is seasonal or spiky, you should skip testing. Sure, you can do some things to mitigate these problems, but a lot of times, you just can’t test because the signal is buried in the noise.

Look instead at whatever qualitative insights you can get from your visitors, other kinds of user research, or best practices.

If you want to understand the full conversion ecosystem, check out the Conversion Ninja Toolbox - a directory of all things conversion-related. Currently, we have about 600 companies in the conversion ecosystem where test and improve is just a tiny part of what you should be aware of. There’s still analytics, usability, user testing, effective copywriting, abandonment recovery, retargeting, and online and offline business integration.

Recent Blog Posts:

Are Tablets Hurting Your Site’s Usability?

Posted by Tim Ash on 12 December 2014 | Comments

Tablets are, in many ways, a nightmare for a lot of retailers.

comments | Read the full post

3 Ways to Make Your PPC Efforts Pay Off

Posted by SiteTuners on 10 December 2014 | Comments

Pay-per-click (PPC) experts Maria Perilli and Shane Ryans hit the nail on the head when they said that it’s futile to be driving traffic to your site all day if your landing page isn’t set up to convert.

comments | Read the full post

Why Visual Design Should Come Last: 6 Steps to a Conversion-Focused Redesign

Posted by SiteTuners on 8 December 2014 | Comments

Everyone has a visual opinion because 90% of the information coming to our brain is visual, and 50% of our brain hardware is designed to process visual information.

comments | Read the full post

FREE Webinar: How to Leverage Visitor Behavior and Social Preferences to Drive Conversion

Posted by SiteTuners on 1 December 2014 | Comments

People are hard-wired to be tribal. That’s not a knock on us - that ability to lock onto a community has served us pretty well. That predisposition is something marketers can tap into to gain significant advantages.

comments | Read the full post

How Tactical Testing Can Be Unhealthy for CRO

Posted by SiteTuners on 30 November 2014 | Comments

There’s nothing wrong with testing, per se. There’s no doubt that it should be part of every content management system (CMS) – not building in the ability to optimize in your CMS seems retrograde. Testing, however, is not equal to conversation rate optimization (CRO).

comments | Read the full post

How to Help Ecommerce Visitors Recover With 404 Page Design

Posted by Tim Ash on 26 November 2014 | Comments

By the time users get to your error page, you’re already struggling about goodwill.

comments | Read the full post

How the Canadian Anti-Spam Law Affects You – And What You Can Do to Keep Your E-mail List Clean

Posted by SiteTuners on 24 November 2014 | Comments

The marketing world’s abuzz with talk of how apps are killing the e-mail channel. However, Craig Swerdloff, Senior Vice-President of E-mail at Experian Data Quality Solutions and CEO of LeadSpend, says this is overkill. E-mail is healthy and robust, is still used as the primary communications channel, and isn’t going...

comments | Read the full post

Feedback and Usability: 3 Ways to Get Visitors to be (a Little) More Patient

Posted by SiteTuners on 17 November 2014 | Comments

Online visitors have the patience of a lit match, but there are dozens of ways to deplete their patience even faster. And while the obvious solution is to speed up your web site, the fact is, world-class speeds are not always practical - mirrored servers can break your budget, minified...

comments | Read the full post

How User Research Can Help Your Conversion Process

Posted by SiteTuners on 14 November 2014 | Comments

Alfonso de la Nuez, UserZoom co-CEO, notes that user experience is about aligning what you’re building internally with what users generally need. With most users on multiple devices, this has gotten more complex. Just getting navigation and search right for a product can be cumbersome, especially with designers and developers...

comments | Read the full post