Tag Archive | "Test"

Marketing 101: What is an A/B split test?

A/B split testing is a powerful way to improve marketing and messaging performance because it enables you to make decisions about the best headline, ad copy, landing page design, offer, etc., based on actual customer behavior and not merely a marketer’s opinion. Let’s break down the process.
MarketingSherpa Blog

Posted in IM NewsComments Off

How a Famous Robot Test Can Help You Beat Impostor Syndrome

Have you ever had that nightmare where you’re sitting in an examination room in front of a panel of experts, watching a timer count down to zero? You’re being asked a series of critical, complex questions, and you’re running out of time to answer. In fact, you haven’t answered one correctly, or at all, and
Read More…

The post How a Famous Robot Test Can Help You Beat Impostor Syndrome appeared first on Copyblogger.


Posted in IM NewsComments Off

How To Craft Content That Stands The Test Of Time (And Grows Your Email List On Autopilot)

 [ Download MP3 | Transcript Coming Soon | iTunes | Soundcloud | Raw RSS ] Have you ever heard the advice, spend 20% of your time creating content and 80% of your time marketing it? In recent years I’ve dished out this wisdom as well, but it’s actually dangerous…

The post How To Craft Content That Stands The Test Of Time (And Grows Your Email List On Autopilot) appeared first on Entrepreneurs-Journey.com.

Entrepreneurs-Journey.com by Yaro Starak

Posted in IM NewsComments Off

SearchCap: Apple search ads, Google’s newest search features & latest carousel test

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.

The post SearchCap: Apple search ads, Google’s newest search features & latest carousel test appeared first on Search Engine Land.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in IM NewsComments Off

Google Test Card Style Local Finder In Maps Results

Mike Blumenthal spotted Google testing a new format for the local finder results in the map results. Instead of a list view…

Search Engine Roundtable

Posted in IM NewsComments Off

SearchCap: Google voters, CPC prices & sitelinks test

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.

The post SearchCap: Google voters, CPC prices & sitelinks test appeared first on Search Engine Land.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in IM NewsComments Off

Does Your Copy Pass the ‘Forehead Slap’ Test?

how to craft compelling copy

One of the most repeated rules of writing compelling copy is to stress benefits, not features.

In other words, identify the underlying benefit that each feature of a product or service provides to the prospect, because that’s what will prompt the purchase.

This is one rule that always applies, except when it doesn’t.

We’ll look at the exceptions in a bit.

Fake benefits

The idea of highlighting benefits over features seems simple. But it’s often tough to do in practice.

Writers often end up with fake benefits instead.

Direct response copywriter Clayton Makepeace asserts that fake benefits will kill sales copy, so you have to be on the lookout for them in your writing. He uses this headline as an example:

Balance Blood Sugar Levels Naturally!

That sounds pretty beneficial, doesn’t it? In reality, there’s not a single real benefit in the headline.

True benefits

Makepeace advises to apply his patented “forehead slap” test to see if your copy truly contains a benefit for the reader. In other words, have you ever woken up from a deep sleep, slapped yourself in the forehead, and exclaimed “Man … I need to balance my blood sugar levels naturally!”

I think not. So getting someone to pull out their wallet to buy that so-called “benefit” will be difficult at best.

Here’s how Makepeace identifies the real benefit hidden in that headline:

“Nobody really wants to balance their blood sugar levels. But anyone in his or her right mind DOES want to avoid the misery of blindness … cold, numb, painful limbs … amputation … and premature death that go along with diabetes.”

A high-risk person will want to avoid the terrible effects of diabetes. That is the true benefit that the example product offers.

How to extract true benefits

So, how do you successfully extract true benefits from features? Here’s a four-step process that works:

  1. Make a list of every feature of your product or service.
  2. Ask yourself why each feature is included in the first place.
  3. Take the “why” and ask “how” does this connect with the prospect’s desires?
  4. Get to the absolute root of what’s in it for the prospect at an emotional level.

Let’s look at a product feature for a fictional “read later” app.


“Contains an artificial intelligence algorithm.”

Why it’s there:

“Adds greater utility by adapting and customizing the user’s information experience.”

What’s in it for them:

“Keeps the data you need the most at the forefront when you’re in a hurry.”

Emotional root:

“Stay up-to-date on the things that add value to your life and career, without getting stressed out from information overload.”

Getting to the emotional root is crucial for effective consumer sales. But what about B2B prospects?

When features work

When selling to businesses or highly technical people, features alone can sometimes do the trick. Overtly pandering to emotions will only annoy them.

Besides, unlike consumers (who mostly “want” things rather than “need” them), business and tech buyers often truly need a solution to a problem or a tool to complete a task. When a feature is fairly well-known and expected from your audience, you don’t need to sell it.

However, with innovative features, you still need to move the prospect down the four-step path. While the phrase “contains an artificial intelligence algorithm” may be enough to get the tech-savvy reader salivating, he’ll still want to know how it works and what it does for him.

The “What’s in it for me?” aspect remains crucial.

For business buyers, you’re stressing “bottom line” benefits from innovative features. If you can demonstrate that the prospect will be a hero because your CRM product will save her company $ 120,000 a year compared to the current choice, you’ve got an excellent shot.

While that may seem like a no-brainer purchase to you, you’ll still need to strongly support the promised benefit with a detailed explanation of how the features actually deliver.

Remember, change can be scary to the business buyer, because it’s their job or small business on the line if the product disappoints.

Sell with benefits, support with features

We’re not as logical as we’d like to think we are.

Most of our decisions are based on deep-rooted emotional motivations, which we then justify after the fact with logic. So, first help create the emotional desire, then aid the rationalization process with features and hard data so that the wallet actually emerges.

Are you a writer who wants to become a Certified Content Marketer?

Inside our Content Marketer Certification program, we’ve got a lot more for writers.

We designed this program to help writers make the most of their careers — to help them position themselves and their offerings, so that they can build profitable freelance writing businesses.

And we’re opening the program soon. Drop your email address below and you’ll be the first to hear about it.

Find out when our Certified Content Marketer training program reopens:

Editor’s note: This post was originally published on February 19, 2014.

The post Does Your Copy Pass the ‘Forehead Slap’ Test? appeared first on Copyblogger.


Posted in IM NewsComments Off

Can You Spot the Expert? Test Your Knowledge of Google’s Content Quality Standards

the best way to get search engines to EAT (and serve) your content

Want to hear something scary? No, not scary like Five Nights at Freddy’s. More like disturbing. Alarming. Even depressing.

I used to write articles about:

  • How to protect yourself from necrotizing fasciitis
  • How to escape from an airplane safety slide
  • How to tell if you’ve been poisoned by sushi
  • Whether runners could benefit from platelet-rich plasma surgery
  • How much alcohol you should drink
  • Why the rate of concussions is higher among women

Now, what makes this admission scary is that I’m not a surgeon. And I’m not a nurse practitioner, physical therapist, or chiropractor.

In fact, I’ve never had any medical training in my life — nor have I ever slid down an airplane safety slide!

Horrified yet? Well, just wait. Because medical advice was not the only thing I used to freely dispense as a web writer.

I used to write articles about child injury law, start-up culture, buying an apartment in New York City, and so on. And I have absolutely no training, experience, or knowledge in any of those areas.

But what’s the big deal, you say? Journalists write about topics they’re not experts in all the time. They simply craft a story from expert sources and authoritative studies. What’s wrong with that?


However, the difference between what I was doing and what a journalist does is that I hardly had time to spell-check, let alone hunt down actual experts, studies, or statistics. Who would when you need to crank out 5 to 10 of these 500-word articles each week?

Sadly, the only knowledge I had was what I found online about these topics. Ah, the glory days of ghostwriting.

Uh, so what exactly makes an expert … an expert?

I wasn’t the only one creating this stuff.

Hundreds (thousands perhaps, maybe even millions) of more drones just like me were clogging up the Internet with shallow, water-thin content on every subject known to man … all in service to people who wanted to game search engines.

Fortunately, Google has since put the kibosh on such behavior through updates like Panda, Penguin, and Hummingbird. And, fortunately, they continue to refine those algorithms, most recently with what they call “Expertise-Authoritativeness-Trustworthiness (E-A-T)”.


That’s Google’s shorthand for what it takes to create high quality web pages and websites. As written in their Search Quality Rating Guidelines, released November 19, 2015:

“High quality pages and websites need enough expertise to be authoritative and trustworthy on their topic.”

These terms — particularly authoritativeness and trustworthiness — are not new to any regular readers of Copyblogger. But have you ever wondered what exactly an expert is?

In some cases, it’s easy to define an expert. For instance, the only person giving advice about knee surgery should be an orthopedic surgeon. Someone with the right training, the proper credentials.

But, according to Google, this is not the only type of expert. Pay attention, because you and I have got something at stake here.

Let me explain.

The rules behind the quiz

I don’t have a college degree in copywriting or content writing.

But because I produce those types of writing for a living — as well as evaluate applications for Copyblogger’s Certified Content Marketers program — it could be argued that I’m an expert.

And you, dear content marketer, are probably struggling with the same type of concern: what exactly makes you an expert?

Well, that’s what this quiz is all about. It’s designed to help you refine your sense of becoming an expert.

Before we get started, let me outline the rules:

I’m going to give you a scenario involving a so-called expert. Your job is to decide if the person described in the scenario is an expert or not.

After each scenario, I’ll tell you the correct answer — according to Google’s content quality standards — and go on to explain the reason behind the answer.

And just so we are clear: every single scenario I share below is a work of fiction, based loosely on real-life experience. But names, places, and incidents are the products of my imagination. Any resemblance to actual persons (living or dead), businesses, companies, events, or locations is entirely coincidental.

Now that we’ve got that out of the way, let’s get going. Ready?

1. Advice about a sports injury

Third-year University of Georgia, Athens economic student and ultra-marathon runner Heather Soso got tired of her chronic plantar fasciitis, a condition she’d been ignoring since her senior year in high school.

Naturally, she did what we all do when we want medical advice: she looked it up online.

She was amazed at the variety of amateur and professional advice available on treating and preventing the condition. Each approach might have some scientific support, but it was mostly anecdotal.

Which approach should she try? It was so confusing! But then she had a brilliant idea: she would try them all and blog about it.

Over the next year, she tried each approach and wrote dozens of articles. Her most popular page was about the six toe exercises that treated her condition successfully.

That’s right: six exercises for her little piggies.

So, what do you think: would Google consider Ms. Soso an expert? Her article on toe exercises authoritative? Trustworthy?

The answer is “yes,” because while her website’s topic is medical in nature, Google would view Heather as an “everyday expert” — someone with relevant life experience.

And because plantar fasciitis is not a life-threatening condition, Google will “not penalize the person/page/website for not having ‘formal’ education or training in the field.”

And this is true for other activities, such as cross-fit training, passing the GMAT, and even teaching SEO. If you’ve got everyday experience, flaunt it!

2. Retirement advice

Dee Dell, from Big Cottonwood Canyon, Utah, is frustrated to no end over the fact that so many Americans don’t have a retirement plan — and don’t even seem to care.

Furthermore, he believes this is not good for our economic future since this may mean that nearly 40 million people will be dependent upon a government that is already stretched thin.

This professor of business management and partner with MegaMo Asset Management is on a mission to encourage men and women over 40 to start saving — and he’s showing them exactly how to do it.

But because Dee is an impatient, aggressive man, his articles are often brief, rushed, and laced with profanity — but oh so much fun to read because of his passion for the subject!

This allows him to churn out four posts a week, but his company and busy schedule with the school keep him from updating the information in his content.

So, what do you think Google would think of Dee’s pages? Expert enough to be authoritative and trustworthy (since he’s got the credentials)?

It’s more than likely that Dee’s pages may not be of the expert variety despite his credentials. Google is explicit that financial advice should come from expert sources but also that the content “should be maintained and updated.”

That’s something Dee is not doing.

In addition, to improve his pages and be taken more seriously by Google, Dee should write in a professional style, go in-depth (even if this means he publishes only once a week), and have his content edited — possibly even reviewed by a peer as well.

3. Tree house building advice

After winning $ 8,047,882 in the Canadian lottery, former newspaper editor and math teacher Kimball Saddlechurn took it upon himself to scratch an itch he’s had since childhood: mastering the art of building tree houses.

But not just any tree houses — really high tree houses.

In the last 6 years, he’s built 14 multi-room tree houses more than 90 feet above the ground. It’s still not clear whether or not these tree houses are legal, but he could care less since he’s a multimillionaire.

Which got him thinking: $ 8 million may not last forever, so maybe he could pad his retirement nest by flipping his hobby into a source of income.

During a casual lunch of veal limone and rabbit gnocchi, his girlfriend told him about the benefits of content marketing. Intrigued, Kimball washed down his meal with a tumbler of Aultmore of the Foggie Moss, spread his laptop out on his indigo pajama bottoms, and launched a sleek website.

In his blog posts, he goes into great detail about the structure and safety of building a tree house that high off the ground. He offers multiple blueprints and considerations about weather conditions and tree types.

This is important, because there is not only money on the line (it takes thousands to build a tree house of this caliber), but lives as well, which makes this Your-Money-or-Your-Life content. (YMYL, for short.)

So, what do you think: would Google consider Kimball’s pages expert enough, especially given the financial nature (people will be dropping thousands of dollars to build a tree house) and risk to life?

Answer: yes.

The reason is that while Kimball is a hobbyist (a rich one at that), he’s got the right type of experience: 6 years, 14 tree houses, and, most importantly, no one has ever fallen out of a tree.

Besides, Google smiles upon the fact that Kimball writes in-depth articles (with blueprints at various angles to boot).

Now, exactly how much experience he needed before he became an expert is unclear. Was it the eighth tree house or the ninth? Maybe it was the fourth?

Here’s a moral I think you can get out of this: there is no perfect time to get started. As long as you are not dealing with lives and big money, you don’t have to wait until a certain number of years to launch.

This is equally true for activities like photography, dog sitting, and learning how to play guitar.

Just start publishing because there are advantages to having a website with age.

4. Advice on a forum

Morton Ambledowny Piff loves Quora — the question-and-answer site where community members ask, answer, and edit the responses. Morton particularly loves sharing answers about his speciality: North Korean culture.

So, it may come as no surprise that this 72-year-old widow and ex-Marine, who spent 37 years working for the government-run Foreign Languages Publishing House in North Korea as a publicist (his fluency in six Asian languages was a major boon), has one of the most popular posts on Quora.

In fact, the article — along with several others — are among the top-ranked in Google search results for a specific keyword phrase. But these top-ranked posts from Morton are not about North Korean culture; they’re about stage IV lung cancer.

You might be thinking, “Huh? How could a former North Korean publicist give medical advice on such a complicated medical topic? Shouldn’t YMYL content come from a medical professional?”

It depends.

See, Morton not only had the unfortunate experience of caring for a father who died of stage IV lung cancer, but Morton himself now suffers from stage IV lung cancer. And his Quora answers are all about his personal experience with lung cancer.

So would Google consider these posts authoritative? This is what Google writes:

“In fact, some types of information are found almost exclusively on forums and discussions, where a community of experts can provide valuable perspectives on specific topics.”

As long as Morton writes about living with and caring for someone with stage IV lung cancer, Morton is an “everyday expert.”

To some degree, he might even be able to write authoritatively about prevention and treatment, but those subjects should probably come from medical professionals.

5. Lifestyle advice

The 33-year-old Wiga Mikolajczak-Jefferson, usually one to agonize for long periods of time over a decision, knew the moment she laid eyes on Blake “The Mighty Thigh” Jefferson that he was her man.

Three days later she was married.

What she didn’t realize was that she’d be moving into Blake’s 251-square-foot bungalow.

But since she was an interior designer by trade and smitten to the bone over her boy, she decided to give it a try. And wouldn’t you know it: after several months of rearranging the bed, she fell in love with the simplicity of living in such a small space.

And because she was a recovering McMansion dweller, she decided to start an email newsletter to tell everyone else about her discovery and the advantages of living a simple, clutter-free life.

Over time, her newsletter attracted 22,000 readers, which made her kind of famous. Unfortunately, though, her blog posts weren’t getting very high search rankings.

Wiga didn’t respond well to this.

“Why are you treating me this way, Google?” she would cry in the dead of the night, shaking her fist.

“Don’t you understand I’m a professional interior designer, have 22,000 readers on my mailing list … and am married to the former NFL running back star Blake Jefferson? Don’t you know that?!”

Sadly, Google ignored her pleas. See, the problem with Wiga’s content boiled down to three things:

  1. Sloppy writing (she refused to capitalize “I”)
  2. Reams of rambling prose (she never got to her point, and when she did, she usually fell down another rabbit hole)
  3. Bunches of broken English

See, according to Google, lifestyle advice falls into the category of “future happiness,” so “advice on parenting issues … should also come from ‘expert’ sources which users can trust.”

And this type of content demands expertise (which she had, both professionally and personally), but it also demands clear, concise, and compelling writing. And it would help to think like a Google engineer, too.

Which, fortunately, means that Wiga can instantly improve the credibility of her content by simply hiring an editor.

A summary of what you should have learned

Let’s wrap this up with some tidy little principles about what we learned, based on section 4.3 of Google’s Search Quality Rating Guidelines:

  • When it comes to high quality medical advice, it “should come from people or organizations with appropriate medical expertise or accreditation.”
  • However, some topics, even medical in nature, only demand that you are an “everyday expert.” Google writes, “These ordinary people may be considered experts in topics where they have life experience.”
  • Aim for deep and detailed content no matter what you write about, but especially if you’re dealing with YMYL content.
  • Perform original research to help your content go deep.
  • Avoid redundant or duplicated content — and don’t steal content from other sites.
  • Edit your content. In other words, spell correctly, fix factual errors, and repair poor grammar.
  • Maintain and update your content on a regular basis.
  • Write in a professional style: clear, concise, and compelling. Be sure to avoid jargon.
  • Remain balanced, professional, and worthy of your audience’s trust.
  • Financial advice should come from expert sources.
  • Cover a topic comprehensively. Don’t aim for an arbitrary word count and stop once you reach it.
  • When giving “future happiness” advice, make sure you have the appropriate expertise (even if it is of the “everyday” variety) and make sure it’s professionally written.
  • Avoid the obvious. If 30 people have already reported on the Facebook Graph Search, then find something else to write about (unless you have information nobody else does).
  • Write content a professional print magazine would publish.
  • Spend an insane amount of time on detail.
  • Commenting on forums like Quora can get you attention and build trust — as long as your posts are encyclopedic, accurate, and easy to read.

Share what you learned in the comments below, and let me know if you have any questions or doubts about whether or not you are an expert.

I know this was somewhat of an unorthodox way to cover this topic, but my hope is that you had fun. Because I know I did.

I look forward to hearing from you.

Want to become a content marketing expert?

Authority is our advanced content marketing education program. Inside Authority, we pull back the curtain on the topics, tactics, and strategies that don’t show up in public blog posts.

The doors to Authority are open until this Wednesday, January 27, 2016, and then we close our doors again until later this year.

Click the button below to join Authority today before the doors close on January 27, 2016.

Join Authority

The post Can You Spot the Expert? Test Your Knowledge of Google’s Content Quality Standards appeared first on Copyblogger.


Posted in IM NewsComments Off

SearchCap: Bing Ads On AOL, SEO Test & The New SMX

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.

The post SearchCap: Bing Ads On AOL, SEO Test & The New SMX appeared first on Search Engine Land.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in IM NewsComments Off

SEO Split-Testing: How to A/B Test Changes for Google

Posted by willcritchlow

At our recent SearchLove conferences, I’ve been talking about things we need to do differently as marketers amidst the big trends that are reshaping search. My colleague Tom Anthony, who heads up the R&D team at Distilled, spoke about 5 emerging trends:

  1. Implicit signals
  2. Compound queries
  3. Keywords vs intents
  4. Web search to data search
  5. Personal assistants

All of these trends are powered by Google’s increasing reliance on machine learning and artificial intelligence, and mean that ranking factors are harder to understand, less predictable, and less uniform across keywords. It’s becoming such a complex system, we often can’t really know how a change will affect our own site until we roll it out. The lack of transparency and lack of confidence in results has two major impacts on marketers:

  1. It damages our ability to make business cases to justify targeted projects or initiatives (or even just to influence the order in which a technical backlog is addressed)
  2. It raises the ugly possibility of seemingly good ideas having unforeseen negative impacts

You might have seen the recent news about RankBrain, Google’s name for the application of some of this machine learning technology. Before that announcement, I presented this deck which highlighted four strategies designed to succeed in this fast-changing world:

  1. Desktop is the poor relation to mobile
  2. Understand app search
  3. Optimize for what would happen if you ranked
  4. Test to figure out what Google wants from your site

It’s this last point that I want to address in detail today — by looking at the benefits of testing, the structure of a test, and some of the methodology for assessing winning tests.

The benefits of A/B testing for SEO

Earlier in the year, the Pinterest engineering team wrote a fascinating article about their work with SEO experiments which was one of the first public discussions of this technique that has been in use on a number of large sites for some time now.

In it, they highlighted two key benefits:

1. Justifying further investment in promising areas

One of their experiments concerned the richness of content on a pin page:

For many Pins, we picked a better description from other Pins that contained the same image and showed it in addition to the existing description. The experiment results were much better than we expected … which motivated us to invest more in text descriptions using sophisticated technologies, such as visual analysis.

– Pinterest engineering blog

Other experiments failed to show a return, and so they were able to focus much more aggressively than they would otherwise have been able to. In the case of the focus on the description, this activity ultimately resulted in almost a 30% uplift to these pages.

2. Avoiding disastrous decisions

For non-SEO-related UX reasons, the Pinterest team really wanted to be able to render content client-side in JavaScript. Luckily, they didn’t blindly roll out a change and assume that their content would continue to be indexed just fine. Instead, they made the change only to a limited number of pages and tracked the effect. When they saw a significant and sustained drop, they turned off the experiment and cancelled plans to roll out such changes across the site.

In this case, although there was some ongoing damage done to the performance of the pages in the test group, it paled in comparison to the damage that would have been done had the change been rolled out to the whole site at once.

How does A/B testing for SEO work?

Unlike regular A/B testing that many of you will be familiar with from conversion rate optimization (CRO), we can’t create two versions of a page and separate visitors into two groups each receiving one version. There is only one googlebot, and it doesn’t like seeing near-duplicates (especially at scale). It’s a bad idea to create two versions of a page and simply see which one ranks better; even ignoring the problem of duplicate content, the test would be muddied by the age of the page, its current performance, and its appearance in internal linking structures.

Instead of creating groups of users, the kind of testing we are proposing here works by creating groups of pages. This is safe — because there is just one version of each page, and that version is shown to regular users and googlebot alike — and effective because it isolates the change being made.

In general, the process should look like:

  • Identify the set of pages you want to improve
  • Choose the test to run across those pages
  • Randomly group the pages into the control and variant groups
  • Measure the resulting changes and declare a test a success if the variant group outperforms its forecast while the control group does not

All A/B testing needs a certain amount of fancy statistics to understand whether the change has had an effect, and its likely magnitude. In the case of SEO A/B testing, there is an additional level of complexity from the fact that our two groups of pages are not even statistically identical. Rather than simply being able to compare the performance of the two buckets of pages directly, we instead need to forecast the performance of both sets, and determine that an experiment is a success when the control group matches its forecast, and the variant group beats its forecast by a statistically-significant amount.

Not only does this cope with the differences between the groups of pages, but it also protects against site-wide effects like:

  • A Google algorithm update
  • Seasonality or spikes
  • Unrelated changes to the site

(Since none of these things would be expected to affect only the variant group).

The statistics and underlying mathematics behind all of this is quite hairy in places, but if you are interested in learning more, you can check out:

Good metrics for measuring the success of tests

We generally advise that organic search traffic is the best success metric for these kinds of tests — often coupled with improvements in rankings, as these can sometimes be detected more quickly.

It is tempting to think that rankings alone would be the best metric of success for a test like this, since the whole point is in figuring out what Google prefers. At the very least, we believe these must be combined with traffic data because:

  1. It’s hard to identify the long tail of keywords to track in a (not provided) world
  2. Some changes could improve clickthrough rate without improving ranking position — and we certainly want to guard against the opposite

You could set up a test to measure the improvement in total conversions between the groups of pages, but this is likely to converge too slowly in practice on many sites. We generally take the pragmatic view that as long as the page remains focused on the same topic, then growing search traffic is a valid goal. In particular, it’s important to note that unlike a CRO test (where traffic is assumed to be unaffected by the test), conversion rate is a very bad metric for SEO tests, as it’s likely that the visitors you’re already getting are the most qualified ones, and doubling the traffic will increase (but not double) the total number of conversions (i.e. there will be a lower conversion rate even though it’s a sensible action).

How long should tests run for?

One advantage of SEO testing is that Google is both more “rational” and consistent than the collection of human visitors that decide the outcome of a CRO test. This means that (barring algorithm updates that happen to target the thing you are testing) you should quickly be able to ascertain whether anything dramatic is happening as a result of a test.

In deciding how long to run tests for, you first need to decide on an approach. If you simply want to verify that tests have a positive impact, then due to the rational and consistent nature of Google, you can take a fairly pragmatic approach to assessing whether there’s an uplift — by looking for any increase in rankings for the variant pages over the control group at any point after deployment — and roll that change out quickly.

If, however, you are more cautious or want to measure the scale of impact so you can prioritize future types of tests, then you need to worry more about statistical significance. How quickly you will see the effect of a change is a factor of the number of pages in the test, the amount of traffic to those pages, and the scale of impact of the change you have made. All tests are going to be different.

Small sites will find it difficult to get statistical significance for tests with smaller uplifts — but even there, uplifts of 5–10% (to that set of pages, remember) are likely to be detectable in a matter of weeks. For larger sites with more pages and more traffic, smaller uplifts should be detectable.

Is this a legitimate approach?

As I outlined above, the experimental setup is designed specifically to avoid any issues with cloaking, as every visitor to the site gets the exact same experience on every page — whether that page is part of the test group or not. This includes googlebot.

Since the intention is that improvements we discover via this testing form the basis for new and improved regular site pages, there is also no risk of creating doorway/gateway pages. These should be better versions of legitimate pages that already exist on your site.

It is obviously possible to design terrible experiments and do things like stuffing keywords into the variant pages or hiding content. This is as inadvisable for A/B tests as it is for your site in general. Don’t do it!

In general, though, whereas a few years ago I might have been worried that the winning tests would bias towards some form of manipulation, I think that’s less and less likely to be true (for context, see Wil Reynolds’ excellent post from early 2012 entitled how Google makes liars out of the good guys in SEO). In particular, I believe that sensibly-designed tests will now effectively use Google as an oracle to discover which variants of a page most closely match and satisfy user intent, and which pages signal that to new visitors most effectively. These are the pages that Google is seeking to rank, and whether we are pleasing algorithms designed to please people or pleasing people directly isn’t too important — we’ll converge on the right result.

What are the downsides?

So, this all sounds great. Why isn’t everyone doing it?

Well, the truth is that it’s quite hard. Not only do most content management systems (CMS) fail to offer the ability to make changes to arbitrary groups of pages, but it’s also hard to gather and analyze the data to come to the right conclusions. There are also theoretical limitations, even on big sites — particularly around understanding and analyzing the effects of changes like internal linking structure, which cascade through the site in unpredictable ways.

We do, however, know of a handful of large sites, with tons of traffic, and huge development resources who have gone down this path and are reaping substantial rewards from it.

Although there will always be sizes of website and levels of traffic below which its uneconomical or impractical to perform sensible tests, we want to make the ability to run these tests available to a much wider audience than it is currently. To achieve this, we’ve been working on our own platform designed both to make it easy to run tests, and also to gather and analyze the output (it also happens to make it easy to roll out quick changes that are hard to get bumped up your backlog, for whatever reason).

Distilled’s Optimization Delivery Network (ODN)

You can read more about the tool in my launch announcement over on the Distilled blog.

As I said over there:

We are calling this type of platform an Optimization Delivery Network or ODN. It works like this:

  • It sits in your web stack like a Content Delivery Network (CDN) (or behind your CDN if you are using one).
  • It allows you to make arbitrary changes to the HTML (and HTTP headers) of any page or group of pages on your website — operating a little like a CMS over the output of your CMS and avoiding the need for a lengthy wait for your development backlog.
  • In addition, it makes it possible to make certain kinds of changes to subsets of pages in order to test to see what will work best.

If you’re interested in hearing more, seeing a demo, or even signing up to the beta, please go ahead and fill out our form.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Moz Blog

Posted in IM NewsComments Off