Get Your Emails in Shape with Multivariate Testing
How is multivariate testing different from A/B testing?
Why should I use it in my email marketing?
How do I do multivariate testing?
Imagine Katie owns Kangaroo Kickboxing, a business that offers martial arts and self defense classes.
Thanks to her great workouts, she’s built up a large email subscriber list that’s made up of current customers and other people who want to learn more about Kangaroo Kickboxing.
Katie decides to open a second Kangaroo Kickboxing on the other side of town so she can increase her customer base. She knows email marketing will be key to getting the word out and filling her new studio with people.
After brainstorming what the email content should be, she narrows it down to a couple different images and calls to action (CTAs).
She’s been playing with different combinations, but isn’t sure which will be the most effective.
Multivariate testing is a lot like A/B testing after it’s gone through Kangaroo Kickboxing strength training.
While A/B testing can only work with one element at a time, multivariate testing can evaluate different, more complex combinations of elements at the same time. For example, let’s say Katie wants to test 2 CTAs with 4 different images.
With A/B testing, she’d first test which CTA worked best. Then, to find out which image worked best, she’d have to test each of them individually with her winning CTA.
Unfortunately, Katie might not get the most reliable results. For example, what if the combination of the losing CTA with the winning image would actually perform the best?
With multivariate testing, she could simultaneously test 8 different versions of her email (2 CTAs x 4 images) to see which combo worked best.
This testing method is fast, effective, and reliable, and can be incredibly valuable to those with large email lists. It even helps you with more than just the current email being tested. It improves all your future emails, too.
That’s because the more of this testing you do, the more you’ll know what works best for successfully marketing to your email subscriber list.
Multivariate testing can help you:
FIND TRENDS
For example, your audience might like drawings over photos.
OPTIMIZE CONTENT
Know how to tweak your copy and design to engage your audience.
STAY FOCUSED
No more guessing and hoping. Instead, create a smart game plan.
LISTEN UP
If A/B testing puts variables into a bracket tournament, then multivariate testing has variables going full fight club on each other. And the first rule of email fight club is: Don’t talk about email fight club. Just kidding, it’s: Choose your variables.
Services like MailChimp help you do multivariate testing, and most let you test up to 8 versions of your emails. That’s a lot of variables you can throw into the ring.
Your variables might be what time you send your email, content, subject line, sender’s name, background color, whether or not you include a photo of your new cat...basically, anything goes. Don’t be afraid to get creative with your testing.
That being said, you should conduct focused, intentional tests based on hypotheses. For example: “I think customers will respond better to direct commands vs. gentle wooing. I’ll test a pushy subject line against a subtler, more enticing one.”
Even if multivariate testing proves your hypothesis wrong, it’s still a win. You’ve just learned something new and valuable about your audience’s preferences.
Okay, you’ve chosen your variables and now you’re ready to test. But wait...did you give your email versions enough room and time to properly compete?
There’s no definitive answer on how many subscribers you need or how long you should run your test. It depends on what you’re testing (opens, clicks, or purchases) and how big of a change you’re looking to find.
In general, you’ll need a large enough testing pool to make sure your test results – and the decisions you make from them – represent a good amount of your audience.
And, you need to run your tests long enough for the right customer engagement data (clicks on links, sales on your website, etc.) to come in.
To help you figure out your testing pool size, you can use “sample size” calculators on sites like Optimizely and Qubit. As for test length, one rule of thumb is to conduct a test for at least 4 hours.
Once you’ve started running your tests, look for meaning in this multivariate world. That is, know what to look for in your results and how to interpret those insights.
On your testing service’s report page, check out your combination results to see which mix of variables is most effective. This can help you discover insights like: “People hate this CTA and this image on their own, but love them together.”
Next, look at results per variable. This tells you how much each individual variable affected your email’s effectiveness across all combinations. That way you’ll know if, for example, your CTA matters more than the image you chose.
Also, know what success looks like. Industry standards say a 20-30% open rate is generally the best indication of a strong email. However, if you’re mainly testing your email’s content, click rate is a good statistic to consider.
DO THIS NOW
In the spirit of testing, let’s figure out how ready you are to do multivariate testing with a short, easy, no-big-deal self-evaluation.
If you’re participating in the course, go to the next section to access your self assessment.
KEY TAKEAWAYS
Multivariate testing lets you evaluate different combinations of variables at the same time.
It can give you valuable insights into your audience's preferences that will help your future email marketing.
To do multivariate testing, choose variables, form hypotheses, have a large email list, and know how to interpret your results.