AB Testing Fails

JitBit posted a very interesting article proclaiming A/B testing doesn’t work.

What is A/B Testing?

“A/B testing is a methodology in advertising of using randomized experiments with two variants, A and B, which are the Control and Treatment in the controlled experiment.” – Wikipedia

Basically, if two people load the same webpage each will see something different on that page.

This could be as simple as one image or word is different or a button’s color is different.

Or it can be as complex as two people see completely different websites.

Everyone’s Doing It

Every major internet business, Google, Amazon, etc… use A/B testing to determine the most effective marketing and design for their sites.

Google AdWords even allows people to do A/B testing on their Pay Per Click advertisements to see which title, text, or URL perform the best.

A/B Tests Always Show the Same Results?

JitBit’s experience was, no matter what they changed, the conversion rate – number of people who purchased – stayed the same.

Fixes vs Scroll Call to Action

JitBit texted having a Call to Action button either remain in one position, or having it scroll with the user as they moved down the page. They found zero difference.

[custom_image src=”https://ellisbenus.com/wp-content/uploads/2013/06/JitBit_fixed_scroll_test-1024×214.png” width=”612″ height=”128″ crop=”” title=”JitBit Fixed vs Scrolling Test” link=”https://ellisbenus.com/wp-content/uploads/2013/06/JitBit_fixed_scroll_test-1024×214.png” lightbox=”lightbox” group=”jitbitabtest” hover=”1″ target=”0″ image_alignment=”” frame=”shadow”]

Low to High or High to Low Pricing

JitBit also tested showing Pricing default ordered from High to Low and Low to High.

They again saw identical conversion data.

[custom_image src=”https://ellisbenus.com/wp-content/uploads/2013/06/JitBit_pricing_test-1024×201.png” width=”612″ height=”120″ crop=”” title=”JitBit Pricing Test” link=”https://ellisbenus.com/wp-content/uploads/2013/06/JitBit_pricing_test-1024×201.png” lightbox=”lightbox” group=”jitbitabtest” hover=”1″ target=”0″ image_alignment=”” frame=”shadow”]

Complete Design Difference

They even went so far as to serve up two completely different homepage designs!

Again, they saw ZERO difference in their conversion rates.

JitBit Old Design

[custom_image src=”https://ellisbenus.com/wp-content/uploads/2013/06/JitBit_old_design-1024×821.png” width=”612″ height=”491″ crop=”” title=”JitBit Old Design” link=”https://ellisbenus.com/wp-content/uploads/2013/06/JitBit_old_design.png” lightbox=”lightbox” group=”jitbitabtest” hover=”1″ target=”0″ image_alignment=”” frame=”shadow”]

JitBit New Design

[custom_image src=”https://ellisbenus.com/wp-content/uploads/2013/06/JitBit_new_design-908×1024.png” width=”612″ height=”491″ crop=”” title=”JitBit New Design” link=”https://ellisbenus.com/wp-content/uploads/2013/06/JitBit_new_design.png” lightbox=”lightbox” group=”jitbitabtest” hover=”1″ target=”0″ image_alignment=”” frame=”shadow”]

Is A/B Testing Dead?!

This Web Developer and Marketer’s opinion is NO.

Google is world renowned for never making decisions without A/B testing nearly everything in their applications.

You Cannot Manage what you Do Not Measure.

Again, the above A/B tests were not failures, rather, JitBit has learned that these elements don’t make a difference to their customers. That in itself is GREAT information! Now, onto the next A/B test! 🙂

Original Article: http://www.jitbit.com/news/185-most-of-your-abtests-will-fail/

Let me about know your A/B testing experiences in the comments!